Efficient Audio-Visual Speech Separation with Discrete Lip Semantics and Multi-Scale Global-Local Attention
Authors: Kai Li
Gao Kejun
Xiaolin Hu
Venue: ICLR 2026 Poster (2026)
Keywords: Audio-video speech separation
vector quantization
lightweight network
discrete semantic units
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Audio-visual speech separation (AVSS) methods leverage visual cues to extract target speech and have demonstrated strong separation quality in noisy acoustic environments. However, these methods usually involve a large number of parameters and require high computational cost, which is unacceptable in many applications where speech separation serves as only a preprocessing step for further speech processing. To address this issue, we propose an efficient AVSS method, named **Dolphin**. For visual feature extraction, we develop **DP‑LipCoder**, a dual‑path lightweight video encoder that transforms lip‑motion into discrete audio‑aligned semantic tokens. For audio separation, we construct a lightweight encoder–decoder separator, in which each layer incorporates a global–local attention (GLA) block to efficiently capture multi-scale dependencies. Experiments on three benchmark datasets showed that Dolphin not only surpassed the current state-of-the-art (SOTA) model in separation quality but also achieved remarkable improvements in efficiency: over 50\% fewer parameters, more than 2.4$\times$ reduction in MACs, and over 6$\times$ faster GPU inference speed. These results indicate that Dolphin offers a practical and deployable solution for high-performance AVSS in real-world scenarios. Our code and demo page are publicly available at https://cslikai.cn/Dolphin.

Published as a conference paper at ICLR 2026
EFFICIENT AUDIO-VISUAL SPEECH SEPARATION WITH DISCRETE LIP SEMANTICS AND MULTI-SCALE
GLOBAL-LOCAL ATTENTION
Kai Lit.2, Kejun Gao-.; & Xiaolin Hu 22233
1. Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China
IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
3. Chinese Institute for Brain Research (CIBR), Beijing 100010, China
{li -24, gkj23J @mails wtwiiis..ssssiiie wwstissiineddiee edu. x xlhuttsinghine en
ABSTRACT
Audio-visual speech separation (AVSS) methods leverage visual cues to extract target speech and have demonstrated strong separation quality in noisy acoustic environments. However, these methods usually involve a large number of pa- rameters and require high computational cost, which is unacceptable in many ap- plications where speech separation serves as only a preprocessing step for fur- ther speech processing. To address this issue, we propose efficient AVSS method, named Dolphin. For visual feature extraction, we develop DP-LipCoder, dual-path lightweight video encoder that transforms lip-motion into discrete audio-aligned semantic tokens. For audio separation, we construct lightweight encoder decoder separator, in which each layer incorporates global- atten- tion (GLA) block to efficiently capture multi-scale dependencies. Experiments on three benchmark datasets showed that Dolphin not only surpassed the current state-of-the-an (SOTA) model in separation quality but also achieved remarkable improvements efficiency: over 50% fewer parameters, more than 2.4 reduc- tion in MACs, and over 6x faster GPU inference speed. These results indicate that Dolphin offers a practical and deployable solution for high-performance AVSS real-world scenarios. Our code and demo page are publicly available at link
mccoer-decoder
INTRODUCTION
In real-world environments, target speech is often masked by background noise and competing talk- ers. This challenge relates to the well-studied "cocktail party effect", where humans can selectively attend to single speech in noisy mixture (Cherry) 1953). Such observations have motivated ex- tensive research on speech separation. However, audio-only speech separation methods frequently exhibit degraded performance in complex acoustic environments. By contrast, the integration of syn- chronous visual cues offers enhanced robustness to noise and has thus motivated extensive research into AVSS methods (Michelsanti et al. 2021).
In recent years, the application deep learning methods to AVSS has attracted considerable atten- tion (Gao & Grauman 2021| Pegg et al. [023 Wu et al.) 2019 Martel et al. 2023 Li et al. 2024b Zhao et al. 2025). Although many methods focus on performance, they often result in computation ally intensive models (such as AV-Mossformer2 (Zhao al. 2025) that
suitable for practical
deployment. Alternative methods like RTFSNet (Pegg et al. 2023) and AVLiT (Martel et al. 2023) aim to improve efficiency using lightweight iterative separators, but the iterative process itself also incurs significant computational overhead. Thus, striking an appropriate balance between separation quality and computational cost remains major challenge in AVSS.
Kai Li and Kejun Gao contribute equally to the article.
1 *Correspnnding author.

---

Published as a conference paper at ICLR 2026
Apart from the computational cost of the audio separator, in the AVSS task, long-standing and even more critical challenge arises from the "path dependence" on video encoders (Martel et al. 2023).
Most methods employ large-scale visual backbone networks that pre-trained on the lip reading task (Gao & Grauman 2021] Pegg et al. 2023 Wu et al., 2019 Li et al. 2024a Zhao et al. 2025
These backbone networks employ larger number of parameters to extract semantically aligned fea- tures from lip movements, resulting in improved speech separation performance. However, this also entails an exceptionally high computational cost for high-performance separation systems. Previous studies have been caught in a dilemma: on the one hand, directly compressing existing large-scale visual encoders often leads to a significant loss in semantic representation capability and a drastic drop in separation performance (Wu et al., 2023); on the other hand, designing lightweight encoders from scratch, tailored for low-level tasks such as video reconstruction, tends to yield only shallow, pixel-level features and fails to extract effective semantic information, resulting in suboptimal sep- aration results (Martel et al. 2023). Therefore, designing video encoders that are both lightweight and capable of sufficient semantic alignment remains major challenge in AVSS research.
To address this issue, we propose a novel and efficient AVSS model, named Dolphin, which aims to improve computational efficiency and separation accuracy. We designed a lightweight dual-path video encoder with vector quantization (VQ) named DP-LipCoder to reduce computational cost while maintaining representational capacity. This video encoder maps the video frames into vi- sual features: one corresponding to compressed abstract visual features that preserve spatio-temporal structures, and the other corresponding to audio-aligned visual features. Specifically, by incorpo- rating VQ (Esser et al. 2021) and leveraging knowledge distillation from a pretrained audio-visual (AV) representation model AV-HuBERT (Shi al. 2022), we map continuous video streams into discrete semantic tokens that are highly aligned with audio.
In addition, we also introduce a lightweight separator with a single-iteration, global-local collabo- rative design. The separator is built upon TDANet (Li et al. (203)). which leverages a dow- attention mechanism. Unlike prior methods that rely on multiple iterations, our approach retains only one forward pass of the separator, while introducing global-local attention (GLA) blocks at each layer to mitigate performance loss. The global attention (GA) block employs coarse- coarse-grained self-attention (CSA) capture long-range dependencies in a low resolution space, whereas the local attention (LA) block adopts heat diffusion attention (HDA), derived from the heat diffusion equation (Cannon 1984). smooth features across channels, suppress noise, and preserve details. With this design, we aim to achieve a balance between computational cost and separation quality.
We validated the effectiveness of Dolphin three public AVSS benchmark datasets (Afouras al., 2018a c Chung et al. 2018). Experiments showed that compared with the SOTA method IANeer Dolphin performed better all separation metries, while also having significant advantages in resource efficiency: more than 50% reduction in parameters, over 2.4x reduction in computational cost, and more than 6x improvement in GPU inference speed. These results indicate that Dolphin offers viable solution for AVSS on edge devices.
2 RELATED WORKS
2.1 AUDIO-VISUAL SPEECH SEPARATION
AVSS methods leverage visual cues like lip movements for robust speech separation in a noisy envi- ronment. Time frequency (TF) domain methods transform speech signals into the time frequennyy representation using the Short-Time Fourier Transform (STFT) for processing (Afouras et al.] 2018b/a Gao & Grauman 2021). End-to-end time-domain models operate directly on raw au- dio waveforms, a direction first pioneered by MU-ConyTTasNe (Wu et al. 2019). Following this line of research, a series of more advanced SOTA architectures such as CTCNet (Li et al., 2024a), IAANet (Li et al.] 2024b) and AV-Mossformer2 (Zhao et al. (225) have been proposed. Despite their improved separation quality, these models are computationally expensive, which limits their applicability in resource-constrained or latency-sensitive scenarios. While lightweight iterative sep- arator reduce parameters, they suffer from inference latency due their recurrent computation (Pegg ce al. 2023 Martel et al. 2023 Sang et al.) 2025). In contrast, we propose a single-iteration en- coder-decoder architecture that, by modeling multi-scale global and local features within each layer, achieves a better balance between computational cost and performance.
2

---

Published as a conference paper at ICLR 2026
2.2 PRETRAINED VIDEO ENCODER
In the AVSS task, the video encoder plays a decisive role in determining overall performance. Ex- isting work typically faces dilemma: On one hand, extract visual features that are well aligned with speech semantics, researchers predominantly rely on large-scale backbone networks pretrained on lip-reading tasks (Gao Grauman, 2021; Pegg et al. 2023] Wu et al. 2019 et al., 2024a bj Zhao et al. 2025). While this strategy leads to improved performance, the substantial computational cost severely limits the model's practical applicability.
the other hand, efforts to reduce this cost have not yielded satisfactory results. Methods such directly compressing these large models or designing lightweight networks from scratch for low-level reconstruction tasks face a common problem: marked decline in semantic representation capability, resulting in poor separation per- formance (Wu et al. 2023] Martel et al. (023). This situation appears to enforce a trade-off rule in the field: performance and efficiency cannot be achieved simultaneously. To solve this issue, propose a novel video encoder. By jointly optimizing semantic extraction and video reconstruction, our method maintains lightweight characteristics while generating features with higher semantic density, thus providing stronger visual cues for speech separation.
METHODS
3.1
OVERALL PIPELINE
A
Audio Encoder
VE
AVF Module
Audio Decoder
Separator
V
Video
V
Encoder
Figure I: The overall pipeline of Dolphin. Frozen parameters are displayed with snowflake marker.
Let S RIXL, and V H xT, represent the input audio and video streams of the target speaker, where La, Tv are the sequence length of the input audio and video separately, and H and W represent the height and width of the video frame. Given a mixture audio A Rlxt, containing utterances of target speaker and C' other speakers B, E Rix along with noise E IRIL,
S Σ +n.
(1)
2
The AVSS task is to recover S from A with the assistance of the video of target speaker V. As shown in Figure the Dolphin consists of five primary components: pretrained video encoder, an audio encoder, audio-visual fusion (AVF) module, separator, and audio decoder.
Specifically, the visual stream V is first fed into a pretrained video encoder. The resulting feature maps are then flattened extract two types of features: reconstruction-relate features V, R.N.X,, and semantics-related features V. E RT,, Here, N, is the dimension of the flattened feature vector, obtained by merging the channel dimension with the spatial dimensions (h, w) ( 20 25 where D represents the number of downsampling. T, denotes the number of temporal frames. For the audio stream, we employ a convolutional layer as the audio encoder to encode the mixture audio A, yielding the audio features X E N.N.T. where N, denotes the number of audio feature channels and T3 denotes the length of audio feature. The visual features Vr. V, together with the audio features X are then fed into the AVF module, producing fused features F E N,X,, Then, we feed into the separator to obtain the target speaker's features E R. N,xT, Finally, the audio decoder employs 1D transposed convolutional layer to transform E into the time-doman signal S RlxL, which corresponds to the separated speech of the target speaker.
W
3

---

Published as a conference paper at ICLR 2026
3D Residual Blocks
Conv3D
Oulput
Residual Blocks
Residual Blocka
Semantic Path
Spellai- attention Block
3D Reeidual Block
AVHUBERT
Lastill
Spatial- étlention Block
Upeamplinng 2x
Upsampling Layer 2x
Upsampling Layer 2x
commnit
Input
Encoder
VQ
Reconstruction Path
Decoder
Feeccoo
Encoder
Input
Conv3D
Resicual Blocks
Residual Blocks
3D Resicual
Blocks
Residual Block
Downsampling Layer 2x
attention Block
Spatial-
Oulpul
attention
Downsampling
Downsampling
Layer
Figure 2: Overall pipeline of the AVDP- WVDP-MagVIT VIT network. The stacked 3D residual blocks denote two 3D residual blocks connected sequentially. The three loss functions are shown as gray blocks.
3.2 PRETRAINED VIDEO ENCODER
the AVSS task, video encoders face a critical trade-off:: large-scale backbones achieve strong semantic alignment with lip movements but incur prohibitive computational costs, while lightweight encoders often fail to capture high-level semantic features and lead to suboptimal separation quality. To overcome this challenge, extracts both reconstruction-related and semantic-related features from the video stream. The dual- path design is motivated by two complementary observations: (i) speaker images inherently contain auxiliary cues such as facial expressions and speaker identity, which are also crucial for speech separation; (ii) speech and lip motions exhibit strong temporal alignment.
design dual-path autoencoder, DP-LipCoder (Figure 2 which
Specifically, both the reconstruction path and the semantic path adopt the same encoder structure. However, The encoder parameters are not shared between the two paths. The encoder structure adapted from video generation network Mag VIT (Yu et al. (023). The encoder consists of cascaded 3D residual blocks and spatial attention blocks, with alternating spatial downsampling. of the semantic path, an additional single-step VQ module is introduced to extract discrete semantic features from semantic path encoder output Z. Subsequently, the outputs of the two encoders denoted as V, and V.. respectively. V, and V, are fused via summation and passed to the decoder, which mirrors the encoder structure. The decoder progressively reconstructs along the spatial dimension, yielding the final reconstructed video V E #XWTT, Further architectural details in Appendix A.
the end
To guide the dual-path network toward simultaneously and effectively modeling V, and V., we introduce three losses during training (Figure 2]: (i) Reconstruction loss (Creom), which improves the fidelity of the reconstructed frames and encourages the reconstruction path to capture speaker- related cues. (ii) Distillation loss (Caisnt), where AV-HuBERT employed guide the semantic path toward extracting audio-aligned semantic features. (iii) Commitment loss (Ccommit) following the standard formulation in VQ (Esser al. (202)) which penalizes discrepan- cies between encoder outputs and their nearest codebook entries. The details of different losses are provided in Appendix A.2 The overall training objective is
teacher model to
L Lcommit Laistill + Creson-
4

---

Published as a conference paper at ICLR 2026
Experimental settings and training results for the pre-training stage are in Appendices A.3 and A.4 respectively. In the AVSS task, we only infer the encoders and their VQ modules corresponding to the reconstruction and semantic paths, thereby obtaining the visual tokens V, and V..
3.3 AUDIO-VISUAL VISUAL FUSION (AVF) MODULE
To achieve efficient AV fusion, the AVF module of Dolphin adopts two fusion mechanisms pro- posed in RTFSNet (Pegg et al. 2023): video-guided gated fusion Fi and attention-based fusion across multiple visual feature spaces F2. Furthermore, we extend these mechanisms from the time- frequency domain to time domain approach. Specifically, we perform upsampling solely along the temporal dimension of the visual features, thereby avoiding redundant expansion in the frequency dimension. Finally, the AVF module aggregates the features obtained from both fusion pathways through element- wise summation, generating the output representation F E RNX., A detailed description of the complete fusion process and implementation can be found in Appendix
3.4 SEPARATOR
GA Block
T.
N₂ x
F
Ma
Encoder Layer
TDA Block
Decoder Layer
Ts
N₂ x
F3
Day
Fs
M2g
Encoder Layer
TDA Block
Decoder Layer
To
N₂ x
F
Day
F2
Mg
Encoder Layer
TDA Block
Decoder Layer
D
Ta N₂
2
TDA Block
Downsample Layer
GLA Block
F
TDA Block
Mi
GLA Block
GLA Block
GLA Block
Encoder Layer
Decoder Layer
N. Tx
F
D
Figure 3: The architecture of separator. Here, set the encoder and decoder layers to 4.
Encoder-decoder architectures have demonstrated strong performance in speech separation tasks
due their ability extract and integrate multi-scale features (Li al. 2023 Xu al. 2025).
Motivated by this, we adopt an audio-only speech separation network TDANet (Li et al. 2023 as
the backbone of our separator, as shown Figure 3 worth noting that the original TDANet
leverages multiple iterative separators to progressively refine the separation results, but this design considerably increases inference time. To improve efficiency, we retain only single separator it- eration while compensating for potential performance degradation by jointly modeling global and local attention in each layer (Shin et al., (0044). This design achieves high-quality separation with substantially reduced computational cost. Specifically, the encoder composed of Q stacked layers, where each layer consists of two global-local attention (GLA) blocks followed by a downsampling layer. At the top layer, employ the GA block within GLA block (see Figure 4(a)). Symmetri- cally, the decoder is built with Q layers, each consisting of a top-down attention (TDA) block from TDANet that performs upsampling, followed by three stacked GLA blocks.
3.4.1 GLA BLOOK
The GLA block (see Figure 4 is designed to enhance audio representations by combining global and local feature modeling. thereby reducing the dependence of the separator on multiple iterations. We describe the first GLA block in the first layer of the encoder, all GLA blocks are identical.
5

---

Published as a conference paper at ICLR 2026
Block
(b) LA Block
Fo
CSA
FFN
F
HDA
FFN
Fn
Bugdusson
LayerNorm
DW/Conw10
DWCoov11
Moon Couuatinn
DWCCoV1DD
Layerlerm
DW/Donr1D
LayerNorm
DWCCom11
F
Fg
010
Folt
DCT
IDOT
GELU
Figure 4: Detailed architectures of GLA blocks in the separator.
GA Block. We design GA block consisting of a CSA layer followed by a FFN. Crucially, the multi-head self-attention (MHSA) mechanism within the CSA layer operates across the entire time dimension of the input sequence. This allows the model to effectively capture global contextual features and long-range dependencies. To mitigate the quadratic complexity of MHSA along the temporal dimension, the CSA layer first downsamples Fo to length T./29, applies a multi-head self-attention function fMHSA to capture global dependencies, and then upsamples the result back to the original length, producing Fo. This design reduces the computational complexity of MHSA to 1 1/220 of the original. Subsequently, the FFN leverages stacked convolutional operations to further refine the Fu, as shown in Figure 4a). [). In particular, depthwise convolution (DWConv ID) with kernel size 3 is employed. The final output is the globally enhanced representation Fo E TR. NoTT
LA Block. Although the GA block demonstrates strong capabilities in modeling long-range contex- tual features, it still exhibits certain limitations in capturing the local contextual structures of audio features. To enhance the model's ss ability to capture local structural patterns, we introduce an LA block, composed of one HDA layer and one FFN, following the GA block (see Figure 4b). The design motivation for HDA layer arises from the following observation: after the feature sequence is projected into sseudo-frequeeee posed into different frequency components (Denis et al. 2002; Imtiaz & Fattah 2013). Therefore, integrating various frequency components allows for precise modeling of local features.
domain via the DCT transformation. local features can be decom-
To efficiently integrate these frequency components, we employ an exponential decay function, de- rived from the heat diffusion equation (Cannon 1984), to implement learnable multi-scale filtering mechanism. Compared with large kernel convolutions, this approach no longer relies on a limited convolutional kernel receptive field and enables fine-grained modeling of local features (see Figure in Appendix), without introducing significant number of parameters. Subsequently, we recon- struct the filtered features and map them back to the time domain via an inverse transform. The key advantage of this design lies in the fact that the basic shape of the filter is constrained by physi- cal priors, and the model only needs to learn a small number of scaling and gating parameters for each channel. As result, the method not only substantially reduces the risk of overfitting but also achieves considerable computational efficiency.
Specifically, we first double the channel dimension of F. through convolutional projection and split the result into an initial condition x E N.NXT, and a gating signal z E N.N.T. We then apply the discrete cosine transform (DCT) Strang 1999) to map x into the frequency domain A(p) E N where p E {0, 1,.., Ta 1} denotes the frequency index:
T--1
(p) = DCT(x)
Σ
(TT(T-0)55 T.
COS
Xy-
In the frequency domain, an adaptive smoothing process is applied to each channel following heat diffusion mechanism. For frequency index p, the attenuation is given by:
A A(p) exp (..))) )) ( E R
(4)
where k. E N. is a learnable, channel-an adaatiie diffusion coefficient controlling the degree of smoothing. The signal is then transformed back to the temporal domain using inverse DCT (IDCT).

---

Published as a conference paper at ICLR 2026
Finally, the output of the HDA block Fo R N.X, is derived through gating mechanism and projection layer:
= O SiLU(Z)),
(5)
where P(.) denotes the multi-layer depthwise convolutional network, and SiLU the activation function Ellwwigg et al., 2018). Finally, FFN identical to that employed in the GA block is applied for feature transformation, yielding the output of the LA block Fo E RNxx2.
3.4.2 ENCODER OF SEPARATOR
The encoder in the separation network employs multiple GLA blocks to efficiently extract and ag- gregate multi-scale features across different temporal resolutions. The core idea is to progressively capture audio-visual representations at different temporal granularities through layer-wise down- sampling. Specifically, after q stages of downsampling, the output features can be denoted as Ti
q [1, .,, Q]} These multi-scale features are further downsampled to the lowest resolution T./24, followed by element-wise summation obtain a global audio-visual rep- T.
{F,
N.x22 2
resentation G E R N, 29 This global representation G is then fed into the top-level GA block,
which enhances its representational power and yields refined global features & E R. N. Sub- sequently, both 9 and the encoder outputs at different scales Fg are input into the TDA blocks (Li et al. 2023). In this process, the global representation serves as guiding signal to modulate local
features, producing new multi-scale features Mg NR-N.x22 To
3.4.3 DECODER OF SEPARATOR
The decoder reconstructs the audio features of the target speaker through top-down connections. Its core is composed of TDA and GLA blocks, whose implementations are consistent with those in the encoder layers. Unlike prior approaches (Li et al. 2024a Zhao al. 2025), decoder directly outputs the target speaker features, without relying on mu multiplying mixture features by a mask (Wang) cet al., 2023). After completing feature extraction from the highest layer to the lowest, the decoder output is passed through an output layer consisting of a 1 x 1 convolution, a GLU activation, and another convolution, yielding the final representation E N.x.
EXPERIMENT CONFIGURATIONS
Dataset. Following previous works (Li et al. 2024a Wu al. 2019 Li et al., 2024b), we conducted a systematic evaluation of the proposed model, alongside several baselines, on three widely adopted AVSS benchmarks: LRS2 (Afouras al.) 2018a), LRS3 (Afouras et al., 2018c), and VoxCeleb2 (Chung et al. 2018). Unless otherwise specified, the training and evaluation uses 2-second speech segments (16 kHz sampling rate), with audio from two speakers mixed by default. Visual and audio are strictly synchronized, with frame rate of 25 FPS, and the visual input consists of 88 x 88 grayscale lip images. More details on the datasets can be found in Appendix
Model Configurations and Evaluation Metrics. We used the Adam optimizer (Kingma & Ba, 2015) for training with an initial learning rate of 1 x 10-3, When the validation loss plateaued for 15 epochs, the learning rate was halved; if it stalled for more than 30 epochs, early stopping was triggered. To prevent gradient explosion, we applied L2 gradient clipping with a threshold 5, and used the SI-SNR (Le Roux et al.) 2019 Appendix[D All training was conducted on 8 RTX 5090 GPUs with a batch size of 48. Detailed hy- perparameter configurations et al. 2006) and SI-SNRi as the main separation metrics, and additionally introduced PESQ (Union 2007 to assess the speech quality. To measure efficiency, we report the model parameters, MACS, CPU/GPU inference latency, and memory usage, all tested on 1-second audio.
the optimization objective, with details provided in
provided in Appendix E For evaluation, we adopted SDRi (Vincent
5 RESULTS
5.1 EPFECTIVENESS OF THE PRETRAINED VIDEO ENCODER
To verify the effectiveness of the Di-Lee, we conducted a separation comparison with pre- trained video encoders on LRS2 dataset (Afouras al. 2018a). Specifically, the baselines include:
7

---

Published as a conference paper at ICLR 2026
(i) 3D ResNet- 18 (Ma et al., 2021) pretrained lip-reading task; (ii) Autoencoder (AE) (Martel
al. 2023) pretrained on single-frame reconstruction task; (iii) LipCoder, a variant of Di-Lip by removing the reconstruction path to examine the effectiveness of the semantic path; and (iv) Dode,,,oddr, incorporating the reconstruction and semantic paths (see Figure 2 For a detailed introduction to AE and 3D ResNet-11. 18, see Appendix
Table I: Comparison of pretrained video encoder methods on the LRS2 dataset. Best results are in bold, with becond-best results underlined. SI-SNRi and SDRi are measured in dB, Parameter count and computational cost (MACs) are calculated for the pretrained video encoder.
Latency (ms)
Methods
SI-SNRi
SDRi
PESQ
Params (MB)
MACs (G/s)
CPU
GPU
3D ResNet-18
17.0
17.1
3,30
11.19
7.95
2908.25
33.10
AE 15.2
15.4
3.15
0.05
0.17
2082.04
22.92
LipCoder
16.3
16.4
3,24
0.65
5.33
2473.84
25.53
Di-Licroder 16.8
16.2
3.29
0.78
2.38
2117.96
23.24
As shown in Table T the results confirm the effectiveness of introducing discrete visual features for AVSS. Both LipCoder and DP-LipCoder leverage the advantages of discrete encoding, achieving clear gains over the continuous autoencoder baseline, with SI-SNRi improvements of at least 1.0 dB. This demonstrates that quantizing visual signals into a discrete "visual vocabulary" encourages more compact and discriminative representations, thereby enhancing separation performance. In addition, the proposed method is highly efficient compared with existing SOTA systems. Taking 3D ResNet-18 as a reference, DP-LipCoder attains a comparable SI-SNRi (within 0.2 dB) while reducing parameters by 93% and MACs by 70%. These findings highlight the strong trade-off between performance and efficiency enabled by our discretization scheme, offering new insights for lightweight yet effective AV system design. Based this balance, adopted the DP-LipCoder for visual feature extraction. Further ablation studies confirm the critical role of the VQ module, which contributes approximately 0.5 dB to the SI-SNRi by enhancing semantic alignment. Detailed results and analysis are provided in Appendix G
Table 2: Performance comparison of AVSS methods with pretrained video encoders replaced by DP- LipCoder on the LRS2 dataset. All results were obtained by retraining the AVSS model from scratch with the frozen Lipderr, Relative changes post-replacement are shown in parentheses, with red indicating improvements and green indicating degradation. Parameter count and computational cost (MACs) represent the combined totals of the pretrained video encoder and AVSS model.
Methods
SI-SKKi 1
SDRi
PESQ
Params (MB)
MACs (G/s) )
CTCNet Li et al. 2024a
14.0, 0.3)
14.4
3.07
7.65,
78.18; 5.57)
RTFSNet (Pea & al. 2023
14.8, -.1)
15.1
3.07
1.63;
10.41
35.94,
5.57)
I1ANet
10.4.
15.6
3.18 -o.05)
4.60;
10.413
-0.94/-..67
AV-Mossformerz Zhao
2025
15.0 0.1)
15.3. 0.2)
3. 3.16
58.50,
10.02)
117.87 6.59)
Generalization to Existing Separation Architectures: To evaluate the generalization ability of
DP-LipCoder, we replaced the original video encoders in several AVSS models with our DP- LipCoder, while keeping the rest of the models unchanged. As shown in Table [ although DP- LipCoder results in slight performance drop, the substantial efficiency gain makes acceptable for practical deployment. In particular, under resource-constrainee scenarios, DP-LipCoder provides an efficient visual encoding solution.
5.2 COMPARISON WITH OF-EE-AR-THEARRY METHODS
We conducted extensive experiments to quantitatively compare the proposed Dolphin with existing methods on three datasets. The results are shown Table 3 For faireess, all baseline scores were either extracted from the original papers reproduced with officially released implementations.
Separation Performance. We conducted comprehensive evaluations of Dolphin' 's separation capa- bilities across three benchmark datasets and compared against SOTA methods. Table demmonssrats that Dolphin consistently surpasses all competing approaches across all evaluation metrics. We fur- ther examined multi-speaker scenarios using the IRS2-3Mix and LRS2-4Mix datasets (Li et al.
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison of various AVSS methods. Results for AV-Mossformer2 and Swift-Net were reproduced using the official code, while results for other methods are reported directly from their original papers.
VoxCeleh2
Methods
SLSNNIi
SDRi
PESQ
SI-SNRi
SDRi
PESQ
SI-SNRi
SDRi
PESQ
AV-Cony WW 2019
Visualvoice
Cas Grammntt
2022
11.3
11.8
2.78
9.9
10.3
2.13
9.3
10.2
2.45
AVLIT-8
Mariel 20A4
12.8
13.1
2.56
13.5
13.6
2.78
9.9
2.23
eareat 2022410
17.5
RTTS Net Pezz B 1116
14.9
15.1
3.07
17.5
17.6
3.25
12.4
13.6
3,00
IIANet [Te 20240
18.5
14.3
NV-ossformer2 Zhab et 2025
15.1
15.5
3.16
17.7
18.1
3,28
14,0
14.6
3.13
Swift-Net
(Sang
13.9
14.2
3.07
15.8
16.4
3.11
12.8
13.5
2.99
Dolphin (ours)
16.8
16.9
3.29
18.8
18.9
3.36
14.6
15.1
3.17
2024b), with detailed results presented in Appendix Dolphin demonstrates consistent improve- ments over existing methods, confirming the generalization capability of our approach. To further validate robustness, we provide evaluations under diverse noise conditions in Appendix [] alongside subjective MOS evaluation on real overlapping speech in Appendix [ which corroborates the su- periority of Dolphin in in-the-wild scenarios. Audio samples in the demo page reveal that Dolphin produces noticeably superior clarity compared bascline methods, particularly under substantial background noise. Additionally, Appendix K provides spectrogram visualizations illustrating the separation performance of Dolphin.
Table 4: Efficiency comparison of AVSS methods. "w/o" and "w/" denote without and with pre- trained video encoder, respectively. "RAM" indicates system memory usage during CPU inference; "GPU Inf." indicates GPU memory usage during inference.
Purameters (M)
MACs (G))
Latency (ms)
Memory (MB)
Methods
w/o
w/
w/o
w/
CPU
GPU
RAM
GPU Inf.
AV-ComvTaasNe (Wa et al.
2019
13.70
25.01
6.94
14.90
342.15
10.54
95.47
129.51
Visualvoice Gao & Gruumam
2021
68.61
77.75
16.82
18.20
280.29
7.19
296.84
338.96
AVLiT-8 Martel al 2023
5.54
5.57
17.52
17.53
547.77
23.31
21.23
$2.09
CTCNet
CecNeeLLidte Li al.. 2024a
6.87
18.06
75.80
83.75
1433.68
37.52
138.42
RIFS-Net Pegg al. 2023
12.03
33.56
4021.93
56.59
IIANet [L
3.82
15.01
18.56
26.51
3213.82
142.30
57.31
148.14
AV
WV-Mosfommrr! (Zhao et
2025
55.87
68.52
111.79
124.46
3083.86
62.30
262.45
398.76
Swift-Net Sang el al.
20255
.147
12.66
18.24
26.18
2589.76
39.45
48.33
244.53
Dolphin (owrs)
6.22
7,00
8,51
10.89
2117.96
33.24
26,84
251.12
Separation Efficiency. Computational efficiency is crucial for real-world deployment of AVSS models. Unlike prior studies that overlooked computational overhead of pretrained video encoders, we comprehensively evaluated both audio and visual components. Table 4 demonstrates that Dol- phin achieves superior efficiency while maintaining strong separation quality. Compared to SOTA IANeet Dolphin required only 46% of the MACs (excluding pretrained video encoder), and its CPU inference time was reduced 66%. Against lightweight alternatives Swift-Net and RTFS-Net. Dol- phin reduced inference time by 18% and 47%, respectively. When incorporating pretrained video encoder, Dolphin achieved the lowest MACs among all methods.
5.3 ABLATION STUDY
To evaluate the effectiveness of each key component in Dolphin, conducted ablation studies the LRS2 dataset, while keeping all other training and evaluation settings identical to the full model.
Contribution of Components in GLA Block. The proposed GLA block enhances audio represen- tations by combining global context and local details. To assess each component's 's role, we evaluated four variants: full GLA (Dolphin), without LA, without GA, and without both (the omitted LA/GA modules are substituted by single depthwise convolution). Results in Table 5 show that removing either part degrades performance, and removing both results the worst outcome. This confirms that global and local modeling are complementary and jointly crucial for effective speech separation.
https://cal iaai .tt..c//oo.oo..iiii......

---

Published as a conference paper at ICLR 2026
Table 5: Ablation study on the contribution of GA and LA blocks in the proposed GLA block.
GA
LA
SI-SNRi
SDRi
PESQ1
Params (MB))
MACs (G/s))
10.4
10.5
2.43
2.04
3.29
15.9
16.1
3.20
5.23
7.59
15.6
15.8
3.18
3.81
6.59
16.8
16.9
3.29
7.00
10.89
Table Performance comparison of model with and without HDA layer.
Methods
SI-SNRi
SDRi
PESQ
Params (MI))
MAC' (G/s))
HDA layer
16.9
16.8
3.29
7.00
10.89
ConviD
16.5
16.4
3.25
7.57
11,19
HDA Layer. To verify the effectiveness of the HDA layer, we compare it with an alternative de- sign where the local feature modeling component is replaced by one-dimensional large-kernel convolution. Specifically, the latter models local features purely through convolution, whereas the former explicitly incorporates physical priors via heat diffusion mechanism. As shown in Table the HDA layer achieves consistent improvements across all three metrics, while also reducing the number learnable parameters during computation.
To further clarify the effectiveness of our design choices, conducted series of ablation studies. First, we compared our single-iteration design against iterative refinement (Appendix D. The results show that our single-iteration Dolphin delivers superior separation accuracy with similar computa- tional cost, while avoiding the overhead of multi-iteration methods, underscoring the efficiency of the GLA module. Building this core design, we examined additional architectural variants. We determined that a relatively deeper decoder leads to improved reconstruction quality (Appendix M and that direct feature regression mitigates distortions from conventional masking (Appendix N Additionally, we confirmed that early-stage audio-visual fusion in the separator's encoder is opti- mal, boosting S1-SNRi by 0.6 dB compared later fusion (Appendix O
CONCLUSIONS
In this study, we propose efficient AVSS model, Dolphin, designed to address the long-standing trade-off between computational efficiency and separation performance in existing approaches. To alleviate the parameter redundancy introduced by conventional visual backhones, we design the DP- LipCoder, which maps lip videos into discrete semantic units that are closely aligned with audio. For the audio network, we design lightweight encoder-decoder separator and incorporate GLA blocks to sequentially capture both global and local features, enabling the separator to achieve high- quality audio separation in a single iteration. Extensive experiments on three benchmark datasets demonstrate that Dolphin achieves a favorable balance between performance and efficiency, offering a practical and deployable solution for AVSS systems real-world scenarios.
L Limitations and Future Work. While Dolphin balances efficiency and performance, several avenues remain for exploration. One limitation lies in the reliance on relatively clean and synchronized lip videos; future work should enhance robustness against large head pose variations, occlusions, and extreme lighting conditions. Additionally, although our model significantly reduces computational costs compared to prior AVSS systems, deployment extremely essorcc-constrainee edge de- vices remains a challenge. We plan to investigate techniques like quantization or pruning to bridge this gap. Finally, while mapping lip motion to discrete tokens improves efficiency, it may discard fine-grained articulatory cues. Exploring hierarchical codebooks or mixed discrete-continuous rep- resentations offers promising direction recapture this subtle visual information.
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China (No. 20212D0200301), the National Natural Science Foundation of China (Nos. U2341220 1228 and
10

---

Published as a conference paper at ICLR 2026
62576187), and the Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Min- istry of Education of China (No. JYB2025XDXM504)
REPRODUCIBILITY STATEMENT
The implementation of Dolphin is developed in Python 3. 1, relying on standard deep learning li- braries, in particular PyTorch and PyTorch Lightning. To ensure full reproducibility, we will release the code of Dolphin under the Apache-2.0 license on GitHub once this work is accepted for publi- cation. The repository will include all necessary files for reproducing the experiments (see Table 3) such as the conda environment specification, complete configuration files, pretrained weights for the video backbone, and the source code of Dolphin. Datasets need to be obtained independently follow- ing the cited references (Appendix [.. they are proprietary to their respective publishers, but will provide preprocessing scripts in the repository. Py PyTorh implementation of VQ, maintained by its original authors, is publicly available on Pype All experiments and training procedures were conducted on a server equipped with eight NVIDIA 5090 GPUs; detailed specifications are reported in Section and Appendix A.33 For researchers aiming to reproduce results, all hyperparameters are listed in Appendix [ Evaluation metrics and loss functions are described with formal definitions in Appendix D and will also be provided in the released codebase
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IEEE Transactions on Pattern Analysis and Machine In- felligence, 44(12):8717-8777, 2018a.
Triantafyllos Afouras, Joon Son Chung. and Andrew Zisserman. The conversation: Deep audio- visual speech enhancement. In Conference of the International Speech Communication Associa- fion. ISCA, 2018b.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Lrs3-ted: large-scale dataset for visual speech recognition, 2018c.
John Rozier Cannon. The one-dimensional heat equation. Number 23. Cambridge University Press, 1984.
Edward Collin Cherry, Some experiments on the recognition of speech, with one and with two ears. Journal of the acoustical society of America, 25:975-979, 1953.
Joon Son Chung and Andrew Zisserman. Lip reading in the wild. In Asian conference on computer vision, pp. 87-103. Springer, 2016.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition, 2018.
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. In Inter- national Conference on Learning Representations (ICLR), 2024.
Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. Fma: A dataset for music analysis. In Proceedings of the 18th International Society for Music Information Retrieval Conference (ISMIR), pp. 316-323, 2017.
Bertrand Denis, Jean Côté, and René Laprise. Spectral decomposition of two-dimensional atmo- spheric fields on limited-ares area domains using the discrete cosine transform (dct). Monthly Weather Review, 150(7):1812-182))) 2002.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
htteass/wwpl.ooo//oooe///teoropp senrantte--ytocch pytorchhcc/ https://callkai..com/m phin
11

---

Published as a conference paper at ICLR 2026
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873- 12883, 2021.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: An open dataset of human-labeled sound events. In Proceedings of the IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 776-780, 2022.
Ruohan Gao and Kristen Grauman. Visualvoice: Audio-visual speech separation with cross- modal consistency. In 2021 IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 15490-15500. IEEE, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: discrimina- tive embeddings for segmentation and separation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 31-35. IEEE, 2016.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and- -nxeitcttiii networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Hafiz Imtiaz and Shaikh Anowarul Fattah. A det-based local dominant feature extraction algorithm for palm-print recognition. Circuits, Systems, and Signal Processing, 32(3):1179-1204, 2013.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Ting- Yu Kuo, Yuanda Liao, Kai Li, Bo Hong, and Xiaolin Hu. Inferring mechanisms of auditory at- tentional modulation with deep neural networks. Neural Computation, 34(11):2273-2293, 2022.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R. Hershey, Sar-half-baked or well done? In 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 626-630. IEEE, 2019.
Kai Li, Runxuan Yang, and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. In The Eleventh International Conference on Learning Represen- fations, 2023.
Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, and Xiaolin Hu. An audio-visual speech separation model inspired by cortico-thallmo-cortiee circuits. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(10):6637-663) 2024a.
Kai Li, Rumxuan Yang, Fuchun Sun, and Xiaolin Hu. lianet: an intra-and inter- miter-yo attention network for audio-visual speech separation. In Proceedings of the 41st International Conference on Machine Learning, pp. 29181-29200 2024b.
Pingchuan Ma, Yujiang Wang, Jie Shen, Stavros Petridis. and Maja Pantic. Lip-reading with densely connected temporal convolutional networks. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 2857-2866, 2021.
Héctor Martel, Julius Richter, Kai Li, Xiaolin Hu, and Timo Gerkmann. Audio-visual speech separa- tion in noisy environments with a lightweight iterative model. In Conference of the International Speech Communication Association. ISCA, 2023.
Daniel Michelsanti, Zheng-Hua Tan, Shi-Xiong Zhang, Yong Xu, Meng Yu, Dong Yu, and Jesper Jensen. An overview of deep-learning-base based audio-visual speech enhancement and separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:1368-1399 2021.
Samuel Pegg, Kai Li, and Xiaolin Hu. Rtfs-net: Recurrent time-frequency modelling for efficient audio-visual speech separation. In The Twelfth International Conference on Learning Represen- fations, 2023.
12

---

Published as a conference paper at ICLR 2026
Wendi Sang, Kai Li, Runxuan Yang, Jianqiang Huang, and Xiaolin Hu. A fast and lightweight model for causal audio-visual speech separation. arXiv preprint arXiv:2506.06669, 2025.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. Learning audio-visual speech representation by masked multimodal cluster prediction. In International Conference on Learning Representations, 2022.
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1883, 2016.
Ui-Hyeop Shin, Sangyoun Lee, Taehan Kim, and Hyung-Minn Min Park. Separate and reconstruct: asym- metric encoder-decoder for speech separation. In Proceedings of the 38th International Confer- ence on Neural Information Processing Systems, pp. 52215-52240 2024.
Gilbert Strang. The discrete cosine transform. SIAM review, 41(1):135-147, 1999.
I. Wideband Union. Wideband extension to recommendation p.862 for the assessment of wideband telephone networks and speech codecs. Recommendation p.862, International Telecommunica- tion Union, 2007.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 14(4): 1462-1469, 2006.
Zhong-Qit -hong-Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. -g-gridnet: Integrating full-and sub-band modeling for speech separation. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:3221-32.: 2023.
Jian Wu, Yong Xu, Shi-Xiong Zhang. Lian-Wu Chen, Meng Yu, Lei Xie, and Dong Yu. Time domain audio visual speech separation. In IEEE automatic speech recognition and understanding workshop (ASRU), pp. 667-673. IEEE, 2019.
Yifei Wu, Chenda Li, and Yanmin Qian. Light-weight visualvoice: Neural network quantization on audio visual speech separation. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pp. 1-5. IEEE, 2023.
Mohan Xu, Kai Li, Guo Chen, and Xiaolin Hu. Tiger: Time-freyuency interleaved gain extraction and reconstruction for efficient speech separation. In The Thirteenth International Conference on Learning Representations, 2025.
Shuang Yang, Yuanhang Zhang, Dalu Feng, Mingmin Yang, Chenhao Wang. Jingyun Xiao, Keyu Long. Shiguang Shan, and Xilin Chen. Lrw-1000, A naturally-distributed large-scale benchmark for lip reading in the wild. In 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019), pp. 1-8. IEEE, 2019.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances neural information processing systems, 2019.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters, (5((0):1499-11) 2016.
13

---

Published as a conference paper at ICLR 2026
Shengkui Zhao, Zexu Pan, and Bin Ma. Clearervoice-studiee Bridging advanced speech processing research and practical deployment. arXiv preprint (rXXi:2506.193900 2025.
APPENDIX
Original signal
Convolution (Gaussian, K-21)
Amplitudo
Heni diflusion (2 00, t-1.2)
0.0
0.4
Time (8)
Original
Convelution
Amplitud
Heat diffusion
0.24
0.26
0.28
0.30
0.32
Time
Figure 5: Comparison of heat diffusion filtering (red dashed, a 0.8, 1.2) and Gaussian convolution (orange solid, K 21) applied to test signal containing multiple frequency components and local impulses (blue dotted). The inset shows magnified view of the impulse response region (green dashed rectangle), revealing superior edge preservation by the heat diffusion method while maintaining effective noise suppression.
A DETAILS OF DP-LIPCODER NETWORK
A. VIDEO AUTOENCODER ARCHITECTURE
In this section, provide detailed description of the implementation of the DP-LipCoder. The decoder is structurally symmetric to the encoder, where each downsampling operation is replaced with the corresponding upsampling operation to reconstruct the features.
ConvaD
ConvaD
SE Attention
Flatten
Selt
RMSNorm
ComvaD
GEGLU
Conv3D
V
G
(a) 3D Residual Block
(b) Spatial-attention Block
Figure 6: Detailed architecture of the 3D residual block and the spatial-attention block.
As illustrated in Figure 2] we design the encoder to project the input video clip V into a com- pact spatiotemporal representation, while alternately performing feature transformation and spatial compression across multiple scales. First, a 3D convolutional layer is applied to expand the input channels, producing V N xT. Unless otherwise stated, the formulas are illustrated us- ing features at the highest spatial resolution; the same computation applies other resolution levels, which differ only in spatial size. At each resolution level, we employ multiple 3D residual blocks (.). to extract spatiotemporal features, shown in Figure 6(a). Each residual unit consists of "two 3D convolutional layers + SE attention + residual connection" and its computation can be formally expressed follows:
14

---

Published as a conference paper at ICLR 2026
(1) Two consecutive 3D convolutions with wose nonlinearity applied as follows:
U ELU (V.)) E ER RRN,.HWXXT
(6)
U ELU Conv3DDxxxI
E ERN.HHHWT.
(2) SE attention (Hu ct al. 2018) (frame-wise spatial aggregation) operates as follows. First, a per-frame x 1 2D convolution is applied to produce the spatial attention logits 4, which are normalized into weights a by spatial softmax. Each channel is then aggregated with these weights to obtain global context vector St. Subsequently, two-layer channel- wise feed-forward network generates the gating coefficients gr. which are broadcast across spatial dimensions to form the channel gating tensor G E RNNx*xxxX procedure can be formalized as:
The entire
1 = Conv2D
U :1 E ERIXHXW t= 1 Te
(8)
(vftmax((ve((.)) E "R
(9)
Sy s.(c) Epen (2( U!? (p,t), c= N,.
(10)
g: of W2 e( Wis))) E R.V.
(11)
Gp(p, (c),, Vp EpeB. EΩ,
(12)
where Ω denotes the set of spatial positions, e(-) is the LeakyReLU and the projection matrices are W1 E Rd...N W2 E NRd,, with hidden dimension d, LN,/2].
(3) We apply channel gating to modulate the intermediate features:
Ô = G O (2) N.NHHXT,
(13)
= U. E
(14)
In this way, a 3D residual block integrates local spatiotemporal modeling with global con- woxte channel recalibration. The residual connection further stabilizes training in deep networks and mitigates degradation.
This residual unit is placed within multi-scale architecture, cooperating with subsequent spatial compression/reoonssctiinn modules.
In the spatial-attention block (see Figure 6 bb)),, we first reshape the visual feature map from the 3D spatiotemporal representation V. Ve E (F Subsequently, E N X into 2D sequence representation
apply the standard self-attention mechanism (Vawwani al. 2017) on this sequence to capture long-range dependencies between different spatial positions within the same time frame, thereby enhancing the model's ability to represent spatial structural information. After the spatial-attention block, we further introduce a feed-forward network to en- hance feature representation capability. This network mainly consists of a normalization layer and a pointwise convolution structure: first, a normalization layer (Zhang & Sennrich 2019) is applied to the input features mitigate distribution differences and stabilize training: then, pointwise 3D convolution layer maps the channel number to an expanded dimension, and a gating unit (GEGLU) (Shazeer (220)) is used introduce higher-order nonlinear representations; finally, another point- wise 3D convolution layer maps the channel number back to the original dimension, resulting in the block output V. TR. XxN.HxWXXX
On the encoder side, we perform progressive compression by spatial downsampling to enhance semantic abstraction. Specifically, each time step the feature tensor is regarded as a 2D image and compressed along the spatial dimensions using a 3 x 3 2D convolution with stride 2, thereby halving the spatial resolution while preserving the temporal dimension T. On the decoder side, a mirror-symmetre structure recovers spatial resolution, where each downsampling step is replaced by sub-pixel upsampling (Shi et al. (2066). This design jointly ensures computational efficiency, spatial detail fidelity, and global consistency during reconstruction.
A.2 OBJECTIVE FUNCTION OF THE PRE-TRAINING STAGE
During pre-training stage, to ensure reconstructability while aligning semantic information, we unify the reconstruction, quantivation-commitmnn. and semantic distillation objectives into single end-
15

---

Published as a conference paper at ICLR 2026
to-end multi-task loss within the VQ-VAE framework. Let V be the input video. The encoder processes V to output continuous latent vectors Ze which are then mapped by a vector quantizer to discrete latent representation V,. Finally, decoder uses V, and V, to produce the reconstructed video V.
Following standard VQ-VAB practice, we adopt symmetric codebook-update and commitment terms, where sg[:] denotes the stop-gradient operator and is the commitment cost (set 1.0):
commii = || Ze] V.ll2 + 3 IZz sg. V.
(15)
where the first term updates the codebook embeddings toward the encoder outputs, and the second encourages the encoder outputs to remain close to the selected codebook entries.
To inject high-level AV semanties from teacher model into the discrete representation, we adopt pretrained AV-HuBERT as the teacher T and introduce a distillation head DD on the post-quantization branch to predict semantic features aligned with the teacher's outputs, yielding the distillation loss:
Lasaill
[ T(V)
(16)
To stabilize training and directly optimize the fidelity of reconstructed lip images, we employ an L2 reconstruction objective:
Ciccon = VIV - VII.
(17)
The overall training objective is
L Ccommit
+
Adisslll Leiistill + Arecon frecom:
(18)
where Adistill controls the relative weight of the distillation signal (set to 1.0 in our experiments), and Arecon controls the relative weight of the reconstruction signal (also set to 1.0).
A.3 EXPERIMENTAL SETUPS
A.3.1 DATASETS
We constructed unified dataset by integrating three publicly available audio-visual corpora: LRS2 (Alouras et al., 2018a), LRS3 (Afouras et al. 2018c), and VoxCeleb2 (Chung et al. 2018). For the raw video streams, 2016), followed by an offline pre-extraction of lip motion sequences. The processed sequences are then segmented into fixed-duration clips. Each clip spans 2.0 frame rate of 25 Hz, yielding approximately N = T' x f = 50 frames. To ensure consistency with prior work (Li et al.) 2024a b), dataset partitioning strictly follows the official configurations of each corpus. Specifically, we merge the original training, validation, and testing subsets into corresponding unified splits, thereby pre- serving the original division of data.
first perform face detection to locate and crop the lip region (Zhang al.,
A.3.2 MODEL CONFIGURATION
The visual codec adopts a lightweight hierarchical architecture composed of residual and attention- based modules. The pipeline comprises stacked 3D residual blocks, spatial-attention blocks, and up/down-sampling layers, connected in the order shown in Figure 2 The input consists of single- channel grayscale, lip-cropped video frames with spatial resolution 88 x 88 and temporal resolution 25 Hz. The channel width starts at 4 and is progressively expanded 32 across stages. All convo- lutions are 3D: the input projection uses a 7 x 7 x 7 kernel, the output reconstruction uses 3 X 3 x 3, and the residual convolutions intermediate layers use kernel size All convolutional layers use constant padding. The attention module uses 8 heads with per-head dimension 32, and FlashAtten- tion (Dao) 2024) can optionally be enabled to further improve the compute and memory efficiency of spatiotemporal modeling.
The discrete codebook is initialized a data-driven manner via k-means. For each quantizer, k- means is run independently 10 times on the encoder features, and the resulting cluster centers are used as the initial entries of that quantizer's codebook. The default codebook size is 256 with embedding dimension 64. During training, stochastic code sampling (temperature 0.1) is enabled to enhance exploration of the codebook space and mitigate the risk of mode collapse due to premature convergence.
16

---

Published as a conference paper at ICLR 2026
A.3.3 TRAINING DETAILS
We optimized Eq. (18) using the Adam optimizer with learning rate of 1 x 10 Training pro- ceeded for up 500 epochs with early stopping: it was terminated if the validation reconstruction loss does not improve for 20 consecutive epochs. Unless otherwise noted, all experiments were conducted on NVIDIA RTX 3090 GPUs using distributed data parallel (DDP). The LipCoder and DP-LipCoder network were implemented in PyTorcc, with global batch size of 32 during training.
A.4 RESULTS
AE
UpCodee DisiipCaceer
Loo (og scdb))
19dx
Figure 7: Reconstruction loss over training steps (log scale; lower is better).
To evaluate the reconstruction performance of our method, trained autoencoder (AE), Lip- Coder, and DP-LipCoder on the same dataset. As shown in Figure 7 the three methods exhibit markedly different convergence behaviors and error floors during training. The LipCoder model without audio alignment remains in a high-loss regime throughout, reflecting codebook competi- tion and drift when cross-modal semantic anchoring is absent; the decoder must compensate for semantic gaps with greater freedom, hindering eтror reduction. The AE trained solely with re- construction loss shows higher floor and sporadic spikes, indicating that its continuous latents primarily capture low-level appearance statistics rather than generalizable audio-visual structure. In contrast, DP-LipCoder unifies reconstruction, quantization-comiiieee into an end-to-end multi-task objective during pretraining, converging with lower variance to the smallest error error-ultimaee ultimately reducing error by about one order of magmitude relative to AE and by more than two orders relative the unaligned VQ. These results indicate that explicitly coupling semantic alignment with reconstruction in a discrete bottleneck improves convergence speed, stabil- ity, and the achievable reconstruction quality, yielding more semantically consistent, reconstructible representations.
and semantic distillation
We further conducted qualitative visualization analysis examine the differences in reconstruc- tion quality across methods. As shown in Figure 8 the misaligned LipCoder model exhibits evident structural omissions and blurry textures in its visual results. In contrast, the AE trained solely with reconstruction loss produces more coherent overall shapes than LipCoder. However, it still strug- gles to preserve fine-grained details and geometric consistency, often resulting in blurred contours or fragmented structures. Our proposed DP-LipCoder model demonstrates clear advantages both glob- ally and locally: it preserves spatial structures more faithfully and generates textures with higher sharpness and realism. Beyond its superiority in quantitative metrics compared to AE and Lip- Coder, DP-LipCoder also achieves qualitatively better semantic consistency and structural fidelity, thereby providing comprehensive evidence of the effectiveness of the proposed approach. Addi- tionally, since LipCoder explicitly encodes lip-motion videos into discrete representations, which better reflect the strong correlation between lip shapes and phonemes, it achieves effective separa-
17

---

Published as a conference paper at ICLR 2026
LipCoder
DP-LipCoder
(a) Demo
AE
LipCoder
DP-LipCoder
(b) Demo
AE
LipCoder
DP-LipCoder
(c) Demo III
Figure 8: Visualization of reconstruction results obtained by different methods (AE, LipCoder, and the proposed DP-LipCoder) on three examples.
tion quality when serving as the pretrained video encoder even though its reconstruction quality is suboptimal.
B DETAILS OF AVF MODULE
Given the visual reconstruction tokens V., semantic tokens V., and audio features X, we first integrate the types of visual tokens with U-Net-based visual feature fusion network (Pegg al. 2023 Li et al. 2023), producing aggregated visual representation V E NTN.T.
18

---

Published as a conference paper at ICLR 2026
For the video-guided gated fusion, the semantic representation V, is processed by 1 x depthwise convolution (..)), followed by temporal upsampling Ø, to generate the gating features. These are combined with audio features Xg. obtained via another 1 x depthwise convolution W2(+), through an element-wise product @. This yields the video-guided gated fusion feature, expressed as
Fi = ===((W((V) & W2(X).
(19)
For multi-visual-space attention fusion, we decompose the visual representation into multiple sub- features, each guiding the fusion with audio features along different dimensions, thereby enhanc- ing the fusion capability. Specifically, wise convolution (-(-) obtain V E (N, This is partitioned into K sub-visual features V& E RR.xxT. The mean of these sub-features is passed through a softmax operation, followed by temporal upsampling, resulting in the multi-space attention representation E RNxxT, Finally, V is fused with audio features processed by Wa(-). yielding
expand the channel dimension of V with
depth-
WV(V) [v V25...;; ;VV]
VA E ERR....
Σ Vk )
V Softmax
#
VER%
(20)
F2 = O (X),
Fg N.XT.
Finally, the AVF module sums the two fused features, producing the output representation F.
C DATASET DETAILS
To comprehensively evaluate the performance of our proposed method, we conducted experiments on three widely adopted benchmark datasets for the AVSS task: LRS2 (Afouras et al. (0188)). LRS3 (Afouras et af., 2018e). and VoxCeleb2 (Chung et al.) 2018). These datasets naturally organize data from different speakers into separate directory structures, which facilitates the construction of speaker-inde tinkkerriddeenneent evaluation paradigms.
To ensure fairness and reproducibility, we strictly followed the dataset partitioning protocols adopted by prior SOTA methods (Pegg et al. 2023 et al. 2024a The detailed characteristics of each dataset are summarized as follows:
LRS2: This dataset is collected from BBC television broadcasts and is characterized by numerous outdoor speaker segments. Visually, illumination conditions vary significantly, while acoustically, the audio often contains authentic background noise, thereby exhibit- ing strong real-world properties. Compared with earlier lip-reading datasets (Yang et al.) 2019 Chung & Zisserman, 2016). LRS2 presents an unrestricted vocabulary and complete sentences, along with higher diversity in speaker identities and recording environments. The mixed version of LRS2 comprises approximately hours training data, 3 hours of validation data, and 1.5 hours of test data.
LRS3: Constructed from over 400 hours of TED and TEDx talks available on YouTube, this dataset benefits from the professional nature of the recordings (e.g., speakers typically use microphones). As result, the audio generally has higher signal-to-noise ratio and a cleaner environment, complementing LRS2. The mixed subset used in our experiments contains 28 hours of training data, 3 hours of validation data, and 1.5 hours of test data.
Derived from large-scale "in-the-wild" YouTube videos, VoxCeleb2 presents highly challenging acoustic conditions, including non-stationary noise such as laughter, background conversations, and variable reverberations, which create rigorous testbed for model robustness. On the visual side, variations head poses, illumination, and image quality make the dataset even more realistic and challenging. The mixed data version consists 56 hours of training data, hours validation data, and 1.5 hours of test data.
VoxCeleb2:
For a fair comparison with existing AVSS methods, we adopted preprocessing pipelines consistent with prior work. For the visual modality, first used FFmpeg to resample all videos to 25 FPS.
19

---

Published as a conference paper at ICLR 2026
A widely used face detection network (Zhang et al. 2016) the lip region, which was resized into a 96 x 96 grayscale sequence. For the auditory modality, FFmpeg used to extract audio tracks from the original videos and resample them into 16 kHz mono signals. During dataset construction, we followed the standard paradigm of mixing clean audio signals from two distinct speakers to generate training samples. While this study focuses two-speaker mixtures, the data generation process is inherently scalable to scenarios involving three or more speakers.
then applied localize and crop
D AVSS OBJECTIVE FUNCTION
We adopt time-domain SI-SNR and frequency-domain SI-SNR (Hershey et al. 2016) as the objec- tive functions to be minimized when training Dolphin. The time-domain SI-SNR defined as
|| S
SI-SNR = 10
(21)
S
S
(22)
STS
where S and denote the ground-truth and the estimated speech signals, respectively.
For the frequency-domain SI-SNR, start from the separator's decoder-layer output D3 (N./87..88) We first upsample along the temporal axis to align Tz/8 to Ta. then apply volution followed by a ReLU activation, dot-multiply with X, and finally pass the result through the audio decoder to obtain the time-domain signal $3 E RIX Let F(.) denote the STFT operator. The magnitude spectra of the ground- truth and estimated speech are given by
M FF()),
= [F(SS)
(23)
where M, M E FXT are the magnitude spectra, is the number of frequency bins, and T the number of frames. Vectorizing the magnitude spectra yields
vec(M) RT m vec(M) FT.
(24)
We define the frequency nonumnn projection coefficient as
m T m w/ m
(25)
and the frequency-domain SI-SNR is
w m 2
SI-SNR((S,5)) = 10
(26)
mm
W m
The overall loss is defined as
C(S Sz) (1 - A) (S,SN + (SL
(27)
where, following a strategy similar to Sepreformer (Shin et al. 2024), we adopt epoch-dependent weighting:
X S0.4,
if epoch 80,
(28)
0.4 gollepcch-80//55, if epoch > 80.
E MODEL. HYPERPARAMETER CONFIGURATIONS
To ensure reproducibility of experimental results and to facilitate future extensions, we provide de- tailed descriptions of the structural configurations and key hyperparameters of each module. Unless otherwise specified, all convolutions oineeimnniional in the temporal domain. The no- tations are defined as follows: convolution kernel size k; stride s, number of attention heads H, per-head dimension d,, feed-forward channel size drEN, and hidden dimension drid-
20

---

Published as a conference paper at ICLR 2026
Visual Encoder. We employ pretrained visual encoder and utilize both reconstruction tokens and semantic tokens as visual inputs. These tokens are fused through element- summation to obtain the visual feature vector, with dimensionality d, 3872. This representation serves the input to subsequent stages of visual feature integration and cross-modal modeling.
Audio Encoder and Decoder. The audio encoder applies one-dimensional convolution (k: 16, 4) to map the raw waveform into a compact latent space, producing a 256-dimensional audio representation. The audio decoder adopts transposed one-dimensional convolution, configured symmetrically to the encoder (A: 16, 8 = 4), in order to progressively upsample and reconstruct the latent representation back the waveform domain.
Visual Feature Fusion Network. The visual feature fusion network in AVF module serves as preprocessing stage for the input video features. We reuse the TDANet module, consistent with RTFSNet, based on the official open-source implementation. The main configuration includes hid- den dimension dhad 64, convolution kernel k 3, upsampling depth of 4, Batch Normalization layers, an 8-head multi-head self-attention (MHSA), and feed-forward network with channel size dFFN 128.
AVF Module. For cross-modal alignment and fusion, this module is constructed with hidden di- mension daia 128, kernel size 1 (1 convolution), an upsampling depth of 4, and Layer Normalization as the normalization strategy.
Separator: The backbone adopts a hierarchical encoder-decoder structure with downsampling layers. In each CSA block, MHSA is configured with H = 8 attention heads and per-head dimension d, 128. In the MSA blocks, we employ a convolution kernel size = 3 and hidden dimension dha 128.
F DETAILS OF DIFFERENT PRETRAINED VIDEO ENCODER
3D ResNet-18: This model first applies 3D convolution to extract temporal and spatial features, followed by using a standard ResNet-18 (He et al., 2016) to obtain visual representations for each frame. The majority of its computational cost comes from the ResNet-18 18 backbone.
AE: A lightweight 2D autoencoder that represents simpler and more computationally efficient approach. The model processes each video frame independently, and its architecture is consistent with that of AVLiT-8 (Martel et al. 2023).
G ABLATION STUDY: EFFECT OF THE VECTOR QUANTIZATION MODULE IN THE DP-LIPCODER
To assess the contribution of the vector quantization (VQ) module in semantic path, conduct an ablation experiment by removing VQ entirely from the visual branch. Specifically, we eliminate the quantizer and its associated commitment loss while retaining the reconstruction and distillation objectives. The modified model is retrained on the same datasets as the original DP-LipCoder, and the separator is trained from scratch with the visual encoder kept frozen during separation training.
Table 7 summarizes the results. Removing VQ leads to a consistent degradation across all metrics, most notably drop of approximately 0.5 dB in SI-SNRi, demonstrating that VQ plays an important role in improving semantic alignment and separation quality.
Table 7: Ablation results for the VQ module on LRS2.
Method
SI-SNRi 1
SDRi 1
PESQ
Without VQ
16.3
16.5
3.27
With VQ
16.8
16.9
3.29
We attribute this improvement to VQ's ability compress continuous features into discrete semantic anchors. which enhances temporal alignment and stabilizes the distillation process against distribu- tion drift. Furthermore, the discrete bottleneck serves as regularizer that suppresses irrelevant
21

---

Published as a conference paper at ICLR 2026
variations, thereby improving robustness to noise and generalization with negligible computational overhead.
H MULTI-SPEAKER PERFORMANCE
Table 8: Performance comparison different methods on LRS2 multi-speaker separation tasks.
LRS2-2Mix
LRS2-3Mix
LRS2-4Mix
Methods
SI-SNRi 1
SDRi 1
SI-SNRi
SDRi
SI-SNRi
SDRi t
U.-ConTTaNNet Wu al. 2019
12.5
12.8
8.2
8.8
4.1
4.6
AVLIT-8 (Martel el al. 2023
12.8
13.1
9.4
9.9
5.0
5.7
CTCNet (Li ct al. 2024a
14.3
14.6
10.3
10.8
6.3
6.9
IAANet ct et al. 2024b
16.0
16.2
12.6
13.1
7,8
8.3
Dolphmn (ou7s)
16.8
16.9
13.1
13.3
9.7
9.9
To ensure a fair comparison, followed the experimental protocol adopted in IIANet (Li et al. 2024b). Specifically, the proposed model was trained and evaluated on the LRS2-3Mix and LRS2- 4Mix datasets. For each dataset containing N speakers, dedicated model was trained handle mixtures with that specific number speakers. During evaluation, we employed the same iterative inference strategy IIANet: for each target speaker in mixture, the model sequentially extracted the corresponding audio stream.
We compared approach against several baseline methods, including AV-ConvTasNe (Wu al. 2019), AVLIT-8 (Martel et al. 2003). CTCNet (Li et al. (2024a), and HANee eLi al. (024b). To ensure fairness, all baseline results were directly taken from IAANet. The quantitative comparison is reported in Table g The results clearly indicate that our proposed method consistently outper- forms all existing baselines across all evaluation datasets in terms of multiple performance metrics. Specifically, our proposed Dolphin achieves the largest improvement over the state-of-the-art on IRS2-44ixx the dataset with the greatest number of speakers, highlighting its superior separation performance in complex real-world scenarios.
I ADDITIONAL EXPERIMENTS UNDER DIVERSE ACOUSTIC CONDITIONS
Although the datasets we used for training and evaluation(c.g.. LRS2, VoxCeleb2) mostly originate from real-world sources(e.g. BBC broadcasts, YouTube videos), we additionally constructed a set of challenging test samples further evaluate the robustness of model. Specifically, we adopted VoxCeleb Chung et al. (2018) the base and introduced environmental noise(from FSD50K Fon- (seca et al. (2022). ), music noise(from FMA Defferrard et al. (2017)) and multiple interfering speak- ers. Finally, constructed four challenging test scenarios based the VoxCeleb2 dataset:
Scenario (Environmental Noise): speaker environmental noise from FSD50K.
Scenario (Music Noise): speaker + music noise from FMA.
Scenario 3 (Environmental Noise Speakers): speaker environmental noise additional interfering speakers (3 speakers total).
Scenario (Music Noise + Speakers): speaker + music noise additional interfering
speakers (3 speakers total).
Each scenario contains 1000 test samples. We evaluated Dolphin along with two represennative base- lines (IIANet and AV-Mossformer2) across scenarios. The results, summarized Table show that Dolphin consistently outperforms the baselines on all metrics (SI-SNRi. SDRi, PESQ), demon- strating greater robustness and separation capability, particularly in the most challenging conditions involving three speakers and additional noise. These experiments complement the main results ported in Section 5 and provide additional empirical support for the effectiveness Dolphin under realistic and more adverse acoustic environments.
22

---

Published as a conference paper at ICLR 2026
Table 9; Performance under four noisy scenarios.
(a) Scenario 1: speaker + environmental noise
Methods
SI-SNRi t
SDRi
PESQ
IAANet
6.69
9.71
2.07
AV-Mossformer2
8.33
9.01
2.13
Dolphin (ours)
11.41
12.36
2.43
(b) Scenario 2: speaker + music noise
Methods
SI-SNRi f
SDRi
PESQ
IIANet
2.04
5.92
1.64
AV-Mossformer2
6.68
7.53
1.76
Dolphin (ours)
7.49
8.52
1.85
(c) Scenario 3: speaker environmental noise interfering
speakers
Methods
SI-SNRi f
SDRi
PESQ
HANet
1.09
4.64
1.44
AV-Mossformer2
3.33
4.71
1.49
Dolphin (ours)
5.37
6.56
1.57
(d) Scenario 4: speaker music noise interfering speakers
Methods
SI-SNRi f
SDRi f
PESQ
HeAet
0.54
3.63
1.36
AV-Mossformer2
3.90
4.56
1.41
Dolphin (ours)
4.11
5.33
1.46
J SUBJECTIVE HUMAN EVALUATION ON REAL OVERLAPPING SPEECH
A large body of prior audio-visual speech separation (AVSS) research is based on datasets such as LRS2, LRS3, and VoxCeleb2, where overlapping mixtures artificially constructed. While these datasets have become standard benchmarks and enable reproducible supervised training. the result- ing mixtures differ from real-worke overlapping speech, where speakers naturally adjust their vocal effort in response to background noise (an effect known the Lombard effect). To better assess practical performance under realistic acoustic conditions, we further conducted Mean Opinion Score (MOS) evaluation on challenging in-the-wild overlapping speech.
We collected 18 short clips (10-20 seconds) from publicly available debate videos that contain nat- ural overlap speech, background noise, and room reverberation. A total of 50 human listeners par- ticipated in the study. Each listener rated the outputs of IIANet, AV-Mossformer2, and our Dolphin using the standard 5-point MOS scale (1 poor, excellent).. The evaluation was performed in double-blind manner with randomized audio order.
Table 10 summarizes the average MOS results across all clips. Dolphin achieves substantially higher scores across all criteria, demonstrating clearer separation and fewer audible artifacts compared to competing baselines. These subjective findings are consistent with qualitative examples our demo page.
K VISUAL. RESULTS FOR SPEECH SEPARATION
We conduct a visual analysis the separation performance of three audio-visual speaker separation (AVSS) approaches: Dolphin, IANett, and AV-Mossformer2. The models are trained the LRS2 dataset, and representative results are shown in Figure 9 To evaluate these methods, we randomly select four audio mixtures from LRS2 test samples.
23

---

Published as a conference paper at ICLR 2026
Table 10: Mean Opinion Score (MOS) results on real overlapping recordings.
Method
MOS
IIANet
2.24 0.15
AV-Mossformer2
2.85 = 0.27
Dolphin (ours)
3.86 0.25
Ground Truth
Dolphin
IIANet
AV- AV-Mosstormer2
Frequency (HX)
Frequancy (H2)
(Hz)
(a) Demo
Ground Truth
Delphin
HANeet
AV-Mossformer2
Frequ anncy (H2)
Frequency (H2)
Frequency (H2)
(b) Demo
Ground Truth
Delphin
HANeet
AV- AM-Mossformar2
Frequamm Jency (He)
Frequanno Leency (H2)
Frequenccy (H2)
(Hz)
(c) Demo
Ground Truth
Dolphin
HANeet
AV- AV-Mossformer2
Frequeenc Junncy (Hz)
Freqquenc uuency (H2)
[H])
(d) Demo
Figure 9: Visualization of separation results obtained by different methods four examples.
In the first sample (Figure 9a Dolphin accurately reconstructs the main harmonic structure of speech, while also producing a cleaner spectrogram with noticeably less background noise. By contrast, the output of ILANet contains more noise artifacts.
In the second sample (Figure 9b). Dolphin again achieves the best performance, with its spectrogram most closely resembling the ground-truth speech. Although AV-Mossformer2 and IIANet are able to separate speech, both methods fail to preserve certain freffueuuncc details.
In the third sample (Figure Sch, Dolphín remains the most effective, retaining the largest proportion of speech components. Results from AV-Mossformer2 and ILANet exhibit significant residual noise, especially in the high-frequency bands.
24

---

Published as a conference paper at ICLR 2026
In the fourth sample (Figure 9d Dolphin once more demonstrates clear superiority by recovering speech with a complete and well-defined harmonic structure. In comparison, both AV-Mossformer2 and IAANet struggle to reconstruct coherent harmonic pattern.
Across all four test cases, Dolphin consistently delivers the best separation results. Its outputs are the most similar the ground-truie spectrograms, with the least noise and the most complete harmonic structures.
L IMPACT OF THE NUMBER OF SEPARATOR ITERATIONS
Table 11: Impact of the number of separator iterations separation performance and efficiency,
Latency (ms))
Methods
SI-SNRI
SDRiT
PESQU
Params (MB).
MACs (G/s).
GPU
AV-TDANet-1
6.4
7.6
2.32
3.89
2.99
676.21
15.80
AV-IDANet-8
12.4
3.89
1185.80
TV 16
12.8
13.2
2,89
3.89
11.68
2981.78
78.30
Dolphin
To validate the effectiveness of the proposed single-iteration design with GLA block, we incorpo- rate the same AVF module as in Dolphin into the baseline audio-only speech separation network TDANet, resulting in AV-TDANet. We compared AV-TDANet with 8 and 16 iterations, specified in the original TDANet paper, against our single-iteration Dolphin. For a fair ablation, we addi- tionally evaluated AV-TDANet with iteration. The construction of AV-TDANet only differs in replacing the Dolphin separator with TDANet. As shown in Table [T] increasing the number of iterations substantially increases the computational burden: MACs grow from 2.99 G/s itera- tion to 7.04 G/s at iterations (2.35x), and further to 11.68 G/s at 16 iterations (3.9x compared to iteration). contrast, Dolphin achieves comparable results with much lower overhead, enabled by the GLA module, which captures both global and local dependencies and ensures favorable lfficiency-performancc trade-off.
M ANALYSIS OF ENCODER AND DECODER ARCHITECTURE
Table 12:
Ablation study on different allocations of GLA modules between encoder and decoder
(total number 5).
Encoder Layers
Decoder Layers
SI-SNRit
SDRi
PESQ
Params (MB)
MACs (G/s))
16.7
16.8
3.29
7.00
10.89
2
3
16.8
16.9
3.29
7.00
10.89
3
16.5
16.6
3.25
7.00
10.89
16.1
16.2
3.24
7.00
10.89
To systematically evaluate the optimal depth configuration of GLA modules within the encoder and decoder, we fixed the total number of GLA modules to 5 and designed several allocation strategies for testing. The primary objective of this experiment is to analyze the relative contributions of the encoding and decoding stages overall performance improvement. The results presented in Table 12 It can be observed that the configuration with GLA modules in the encoder and in the decoder achieves the best performance across all evaluation metrics. In contrast, other configurations fail to reach the same performance level. This finding suggests that relatively deeper decoder plays critical role in accurately reconstructing high-quality target speech from high-level semantic representations, and further underscores the importance of carefully balancing modeling capacity between the encoder and decoder.
N ABLATION STUDY: OUTPUT FEATURE, FORMULATION OF THE SEPARATOR
In order to investigate the optimal representation of the separator output features, we conducted ablation study. Mainstream speech separation approaches are typically mask-based (Wu al.
25

---

Published as a conference paper at ICLR 2026
2019 Li al. 2024a b Pegg al. 2023 Zhao et al. 2025 Sang al. 2025 where the separator generates a feature mask that is applied to the encoder's ' mixture representation through element- wise multiplication to estimate the target speaker representation. In contrast, adopt direct mapping strategy: instead of employing a masking mechanism, the decoder directly regresses the embedding of the target speaker. We hypothesize that directly predicting the target representation can mitigate potential nonlinear distortions or information loss introduced by the masking operation Wang et 2023).
Table 13: Comparison of direct mapping and masking strategies on the LRS2 dataset.
Methods
SI-SNRi t SDRi
PESQ
Mapping 16.8
16.9
3.29
Mask
16.3
16.4
3.20
evaluated both approaches (direct mapping vs. masking) on the LRS2 dataset, and the results are summarized in Table 13 The direct mapping strategy consistently outperforms the masking-basee approach across all objective metrics. Specifically, our method achieves a 0.5 dB improvement in both S1-SNRi and SDRi, while also obtaining additional 0.09 in PESQ. These results strongly demonstrate the effectiveness of direct mapping, suggesting that regressing the target representation directly from the mixture provides a superior paradigm. Consequently, our final model adopts the direct mapping scheme.
O ABLATION STUDY: EFFECT OF FUSION LOCATION
To systematically evaluate the impact of fusion position between visual and auditory features on speech separation performance, we conducted comparative experiments four candidate positions Fo F3, as illustrated in Figure 3 Specifically, Fo denotes the earliest fusion at the beginning of the acoustic encoding process, while F1 F3 progressively inject visual features into deeper stages of speech representations. Apart from the fusion position, all training and inference configurations were kept identical to ensure a fair comparison.
Table 14: Comparison of different fusion positions on the LRS2 dataset.
Fusion Position
SI-SNRi
SDRi
PESQ
Fo Fi
16.8
16.9
3.29
16.5
16.6
3.22
F2
16.3
16.4
3.21
F3
16.2
16.3
3.21
Table 14 reports the separation performance on the LRS2 dataset. As shown, the earliest fusion Fo achieves the best results across all three metrics: compared to the latest fusion F,, yields improvements of +0.6 dB in SI-SNRi. +0.6 dB in SDRi. and +0.08 in PESQ. Moreover, as the fusion position shifts from Fo to 3, the performance consistently declines. This observation aligns with findings in neuroscience regarding cross-modal interactions (Kuo et al. 2022).
P LLM USAGE
In preparing this manuscript, the authors used a large language model (LLM) solely to assist with polishing the wording and improving the clarity of English expression. The LLM did contribute to the research design, data collection, data analysis, or the generation of scientific ideas. Its role was limited to helping refine sentence structure, grammar, and flow of the written text. All substantive content, methods, and conclusions entirely the authors' work.
26


Reviews:
[{"review_id": "NK0VFUTyQ7", "reviewer": "Reviewer_Qs5W", "summary": "The paper presents a more computationally efficient AV speech separation system called Dolphin. As part of this system, the author's also present a lightweight visual encoder that maps video frames to semantic, well-aligned-to-audio discrete tokens. Together they yield state-of-the-art quality across a variety of corpora and metrics while being fairly computationally efficient.", "questions": "How well does the Dolphin system do with real overlapping speakers?\nHave conducted any human listening tests / comparative evals against other systems?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper demonstrates an approach for building a state-of-the-art speech separation system. They modify TDANet, moving from iterative, progressive separation to single separation step for efficiency with joint local and global attention for each layer for a large win on efficiency. They also note the need for improving throughput in the vision encoder, which encodes lip motion frames into discrete semantic tokens, by developing a fast, high quality model called DP-LipCoder. Third, for separation they use an efficient, mixed global/local attention encoder/decoder. These innovations yield a system that surpasses many recent AV speech separation systems.\n\nFurther, the system is efficient showing less memory use, fewer MACs, faster GPU inference than many competing systems. \n\nThe paper will provide a github repo that will allow others to fully reproduce results as much as possible.", "weaknesses": "In some ways, this paper is an incremental approach to AVSS, not providing many new insights, however strong the results are. Critics may wonder if human evaluations/side-by-sides would agree that the Dolphin system is indeed superior to the other approaches in terms of quality improvement and fewer artifacts. Lastly, the tasks all appear to be artificially created speaker overlap and equation (1) is a large simplication--with the Lombard effect the noise causes non-linear changes to speech production.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yyBMBqT9HX", "reviewer": "Reviewer_Pm3b", "summary": "This paper tackles audio-visual speech separation (AVSS). The core problem it targets is that current SOTA methods either achieve good separation quality but rely on a heavy visual backbone plus multiple iterations of the separator, or they are lightweight but require repeated iterations, which hurts real-time performance and deployability. The authors propose an overall framework called Dolphin, with two main components: a lightweight dual-path video encoder DP-LipCoder and a single-iteration encoder–decoder separator with Global–Local Attention.", "questions": "The proposed GLA-based separator is presented as a single-pass design for efficiency. However, since the block itself is a generic feature transform, it seems technically feasible to unroll it for multiple passes (as commonly done in audio-visual separation) to trade extra compute for further gains. I would like to see how the model performs under multiple iterations.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It makes explicit a long-standing but very real issue in AVSS that the visual encoder is too heavy, and then deliberately designs a dual-path + VQ + distillation structure on the visual side. This is a more thoughtful route than simply plugging in an off-the-shelf lipreading backbone or adding a small autoencoder. It jointly considers both reconstructive and semantic aspects, rather than just trimming the backbone.\n\nThe separator also states the efficiency goal of replacing multiple iterations with a single pass quite clearly, and then compensates for it with global and local components.", "weaknesses": "The paper clearly states that compressing lip-reading backbones causes semantic loss, and that purely reconstruction-oriented lightweight encoders only capture shallow, pixel-level cues. It then proposes a dual-path design with VQ-based discrete semantics plus distillation to address this. However, the mechanism of why the dual-path + discrete tokens specifically preserve the task-relevant semantics under heavy compression is mostly justified empirically (Table 1), rather than analytically. A more explicit explanation or diagnostic would strengthen the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IOmYB64DwX", "reviewer": "Reviewer_1MMg", "summary": "This paper presents an audio-visual model for speech separation. The main contribution is a model which achieves SOTA results and is also lightweight. Results on multiple datasets and an ablation study are presented.", "questions": "Please see above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of an audio-visual speech seperation model which achieves SOTA performance and is also lightweights is a useful contribution.\n- Results on multple datasets and detailed ablation study.", "weaknesses": "- The title is a bit misleading. The model recovers the speech of the main speaker only, and not all speech signals at the same time. So speech enhancement might have been a better term for this task (and it's more consistent with the literature).\n- The paper does not compare with some recent SOTA models, e..g. \"LA-VocE: Low-SNR audio-visual speech enhancement using neural vocoders\n- It's not clear what results (in terms of number of background speakers) are presented in Section 5. Given that the appendix includes results with multiple background speakers, then these results probably include 1 background speaker. Also, why not testing the model's performance on different types of background noise and combination of background noise+multiple speakers.\n- No supplementary material is provided, so it's hard to judge how good the denoised samples are. The numbers look good, but without listening to examples it's hard to judge the quality. Some examples are provided on the provided link, but they are too few. There are only 6 examples where the model is compared to other models. Would also be good if some details about the type of noise in each example are added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P2hFzh7Urg", "reviewer": "Reviewer_mwcV", "summary": "The paper introduces Dolphin, a compact and efficient audio-visual speech separation framework that integrates visual motion cues with attention-based processing to isolate target voices. It employs DP-LipCoder, a lightweight dual-path video encoder that learns discrete visual representations aligned with speech, and a single-iteration separator with global–local attention for efficient feature modeling. Through its optimized design, Dolphin achieves superior separation quality with over 50% fewer parameters, 2.4× lower computational cost, and 6× faster inference, making it well-suited for real-time and edge deployment.", "questions": "* It is unclear whether the encoders in the Semantic Path and Reconstruction Path shown in Figure 2 are trained separately or if there is parameter sharing between them. A more detailed explanation would be helpful.\n\n* For the results reported in Table 2, it is not specified whether the performance was measured after retraining the model with DP-LipCoder or simply by replacing the visual encoder without additional training. Clarification on this point is needed.\n\n* Including a discussion on the limitations of this work would make the paper more comprehensive and could provide valuable guidance for future research.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "For the AVSS task, we designed a highly efficient model that outperforms existing approaches in key metrics such as SI-SNRi and SDRi, while achieving a 50% reduction in parameters and 2.4× lower computational cost, effectively balancing the trade-off between performance and efficiency.\n\nAlso, throughout the paper, reasonable architectural designs such as the single-pass GLA mechanism and the DP-LipCoder, which quantizes lip movements into discrete tokens, were employed to reduce computational cost while maintaining high performance. The effectiveness of these architectural innovations is well supported by the experimental results.", "weaknesses": "There is a lack of ablation studies for individual components. In particular, an additional experiment comparing the performance without using VQ, which plays a crucial role in this work, would be valuable.\n\nMoreover, the paper lacks sufficient analysis of the individual contributions of global and local attention within the GLA blocks, as well as comparisons with other audio-visual fusion strategies\n\nIt is also unclear whether Dolphin can maintain its performance without a pretrained teacher model such as AV-HuBERT.\n\n* Minor weaknesses\n    * Line 234: The link for Figure 3 should include the letter “F.”\n    * For better experimental transparency, it would be helpful to indicate which results in Table 3 were taken from the original papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)