SonicSim: A customizable simulation platform for speech processing in moving sound source scenarios
Authors: Kai Li
Wendi Sang
Chang Zeng
Runxuan Yang
Guo Chen
Xiaolin Hu
Venue: ICLR 2025 Poster (2025)
Keywords: Moving audio toolkit
moving audio dataset
speech separation
speech enhancement
Primary area: datasets and benchmarks

Abstract: Systematic evaluation of speech separation and enhancement models under moving sound source conditions requires extensive and diverse data. However, real-world datasets often lack sufficient data for training and evaluation, and synthetic datasets, while larger, lack acoustic realism. Consequently, neither effectively meets practical needs. To address this issue, we introduce SonicSim, a synthetic toolkit based on the embodied AI simulation platform Habitat-sim, designed to generate highly customizable data for moving sound sources. SonicSim supports multi-level adjustments—including scene-level, microphone-level, and source-level—enabling the creation of more diverse synthetic data. Leveraging SonicSim, we constructed a benchmark dataset called SonicSet, utilizing LibriSpeech, Freesound Dataset 50k (FSD50K), Free Music Archive (FMA), and 90 scenes from Matterport3D to evaluate speech separation and enhancement models. Additionally, to investigate the differences between synthetic and real-world data, we selected 5 hours of raw, non-reverberant data from the SonicSet validation set and recorded a real-world speech separation dataset, providing a reference for comparing SonicSet with other synthetic datasets. For speech enhancement, we utilized the real-world dataset RealMAN to validate the acoustic gap between SonicSet and existing synthetic datasets. The results indicate that models trained on SonicSet generalize better to real-world scenarios compared to other synthetic datasets. Code is publicly available at ***https://cslikai.cn/SonicSim/***.

Published as a conference paper at ICLR 2025
SONICSIM: A CUSTOMIZABLE SIMULATION PLAT- FORM FOR SPEECH PROCESSING IN MOVING SOUND SOURCE SCENARIOS
Kai Li!.2, Wendi Sang', Chang Zeng", Runxuan Yang', Guo Chen' & Xiaolin Hu H2
1. Department of Computer Science and Technology, Institute for AI, BNRist, Tsinghua University, Beijing 100084, China
2. Tsinghua Laboratory of Brain and Intelligence (THBI),
IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing 100084, China
3. National Institute of Informatics, Japan
Chinese Institute for Brain Research (CIBR), Beijing 100010, China
s inghua. kai 1 @gmail. com
x slhu@tsinghua edu. en
ABSTRACT
Systematic evaluation of speech separation and enhancement models under mov- ing sound source conditions requires extensive and diverse data. However, real- world datasets often lack sufficient data for training and evaluation, and synthetic datasets, while larger, lack acoustic realism. Consequently, neither effectively meets practical needs. To address this issue, we introduce SonicSim, synthetic toolkit based the embodied Al simulation platform Habitat-sim, designed generate highly customizable data for moving sound sources. SonicSim supports multi-level adjustments-iicludee
scene-level. microphone-level, and source- level-enabling the creation of more diverse synthetic data. Leveraging Sonic- Sim, we constructed benchmark dataset called SonicSet, utilizing LibriSpeech, Freesound Dataset 50k (FSD50K), Free Music Archive (FMA), and 90 scenes from Matterport3D to evaluate speech separation and enhancement models. Ad- ditionally, to investigate the differences between synthetic and real-world data, we selected 5 hours of raw, non-reverberant data from the SonicSet validation set and recorded real-world speech separation dataset, providing reference for compar- ing SonicSet with other synthetic datasets. For speech enhancement, we utilized the real-world dataset RealMAN to validate the acoustic gap between SonicSet and existing synthetic datasets. The results indicate that models trained on Sonic- Set generalize better real-world scenarios compared to other synthetic datasets. Code is publicly available at https: /ttp:::///skkaaaaa cn/ wwriiinn//SSonnn cn/SonicSim/
INTRODUCTION
Speech separation (Cherry 1953) and enhancement (Loizou 2007) are classic tasks focused on ex- tracting target speech from noisy audio. The release of large-scale synthetic datasets like LibriMix (Cosentino et al. (020)) and the DNS Challenge (Reddy et al. 2020) has propelled model advance- ments, with many methods (Chen et al., 2020a Luo Yu 2023b) demonstrating cross-environment transferability in zero-shot settings. These SOTA methods have significantly benefited applications such as online meetings (Rao et al., (02T)) and human-compp comaner interaction (Yu & Chen 2017). However, significant acoustic mismatch between synthetic and real-world data persists, leading to suboptimal model performance in real-world scenarios (Sivaraman Kim 2022).
Previous efforts to bridge the gap between synthetic and real-world data have mainly focused on more accurate modeling of room reverberations. Datasets like WHAMR! (Maciejewski et al., (022)) and the DNS Challenge (Reddy al.] 2000)) simulate room impulse responses (RIRs) using methods such as finite element analysis (Jarrett et al. 2012). ray tracing (Lopez- 2000), or the
"onresponning author.

---

Published as a conference paper at ICLR 2025
image source method (Luo & Yu 2023a) However, from the perspective of RIR simulation, existing approaches face three major challenges: 1) Occlusion obstacles: RIRs based on the image source method struggle with varying quantities and shapes of obstacles, particularly when objects obstruct the sound path between the source and microphone (Aralikatti et al.] 2022). 2) Complex room
geometry: Simulations often assume cuboid-shaped rooms (Southern al. 2011), deviating from complex real-world environments, such as lecture halls with tiered seating. 3) Complicated room surfaces and object materials: Different materials have distinct acoustic properties, and accurately replicating scenes using multiple materials is challenging. Therefore, accurate simulation of RIRs remains difficult problem.
Existing datasets are primarily collected from static sound sources (Cosentino et al. 2020 Ma-
ciejewski et al. 2020; Reddy et al. 2020), making them suitable for non-moving scenarios like online meetings. In contrast, moving sound source trenarios-wher speakers move around while static microphones capture their speech- are more relevant to applications such as robotic naviga- tion (Wang et al. 2004). However, capturing large-scale datasets for moving sound sources is more time-consuming, leading to a scarcity such data, which limits the development and evaluation of speech separation and enhancement models in these tasks. Additionally, real-world datasets constrained by physical environments and, once collected, their distributions are fixed, making them difficult to adjust flexibly and increasing the risk of models overfitting to specific datasets. Thus, despite their high research value, the lack of relevant datasets for moving sound source scenarios significantly hinders deeper exploration in this area.
Scene Import
Acoustic Simulation
Microphone Types
Positions
Functions
Mone
AnD scrica
SonicSim
Figure 1: Overview of SonicSim, our toolkit for speech research. SonicSim provides a customizable data generator based on Habitat-sim, allowing users to generate realistic and physically plausible audio data in controlled manner.
Librispeech
FSDSOK
FMA
SonicSet
90 Matterport3D scenes
FMA music
57,896 motion trajectories,
Figure 2: Overview of the SonicSet dataset. It covers a wide range of scenarios, speakers, and noise.
To advance research in moving sound sources, particularly speech separation and enhancement tasks, there is pressing need for a high-fidelity, low-cost, and comprehensive synthetic toolkit and data asset for moving sound sources. To address these issues, we developed SonicSim, a synthetic toolkit capable of accurately simulating RIRs for moving sound sources, shown in Figure I This toolkit is built upon the embodied AI simulation platform, Habitat-sim (Savva et al. 2019 which can import variety of 3D environments and accurately simulate the acoustic characteristics of rooms. The core strength of SonieSim lies in its precise simulation of RIRs (Chen al.) (002a) and highly customizable simulation of moving sound sources, which significantly expands the scale of data collection for moving sound source research.
Based on SonicSim, we constructed a multi-scene, large-scale, and high-quality moving sound source dataset called SonicSer (see Figure 2h: 1) Multi-scene: SonicSet utilizes 90 scenes from the Matterport3D dataset (Chang et al. 2017 covering a wide range of real-world environments,
such as homes, offices, and churches: 2) -cage-scale: SonieSet integrates 360 hours of speech audio
from the LibriSpeech (Panayotov et al.
2015
combined with environmental noise from FSD50K
2

---

Published as a conference paper at ICLR 2025
(Fonseca et al. 2021) and musical noise from the FMA dataset (Defferrard al. 2017): 3) High- quality: The RIRs of the synthetic audio closely resemble real-world environments by simulating reflection and diffraction across various materials, resulting in higher-quality reverberated audio.
On the SonicSet dataset, we conducted extensive quantitative experiments on 11 speech separation methods and 9 speech enhancement methods, thoroughly analyzing the performance of each ap- proach. To assess the gap between SonicSet and real-world environments, we created a speech sep- aration dataset by recording raw, non-reverberant audio from the SonicSet validation set in real-life scenarios across 10 scenes, totaling 5 hours. For the speech enhancement task, we utilized the Real- MAN test set (Yang et al. 2024), which includes real-world recordings of moving sound sources, to evaluate discrepancies between the synthetic dataset and real environments. The experimental results demonstrated that models trained on SonieSet generalized well to real-world conditions, confirming the effectiveness and potential of SonicSim for speech research.
RELATED WORKS
In this section, we systematically compare the SonicSim toolkit and the generated SonicSet dataset with existing real-world datasets, synthetic datasets, and simulation toolkits (see Table I
Currently, there are very few speech separation and enhancement datasets recorded in real-world environments (Yang et al. 2024 Watanabe al. 2020). This primar- ily because most speech datasets are designed for speech recognition evaluation and only provide transcription annotations, lacking the ground-truth labels needed for supervised learning tasks such as speech separation and enhancement. Recently, the RealMAN dataset introduced real-world mi- crophone array recordings of speech and noise (Yang et al., 2024). Although the gap between the RealMAN dataset and real-world application scenarios is relatively small, its high annotation cost and limited scalability hinder model adaptation to diverse scenarios. Our proposed SonicSim ad- dresses these limitations by offering highly customizable and realistic synthetic data generation method, filling the gap in scale and scene diversity present in existing datasets.
Real-world datasets:
Datasets
Geometry
Occlusion
Material
Scalability
Cost
Tools
Src Type
Tasks
WHAMR 0020
Caboid
x
Low
Static
SS/SE
LibriCSS 20200
Cuboid
x
High
X
Static
SS
DNS Challengs
Cuboid
x
Low
Static
SE
Chimeo 2020
Variable
X
High
X
Statie
SS
LRS2
Variable
X
High
RealMan 2014
Viriable
X
High
X
Dynamic
SE
SonicSel (ours)
Variable
Low
Dynamic
SS/SE
Comparison of different datasets for speech separation and enhancement. "Geometry" refers to the room geometry. "Occlusion" indicates whether objects obstruct the sound source and microphone. "Material" shows whether the material properties of objects are considered. "Scalabil- ity" refers to the ability to increase the amount of data. "Cost" represents the difficulty of simulating or collecting data. "Tools" indicates whether there are toolkits for data collection. "Sre Type" indi- cates the source type. "SS" and "SE" denote the speech separation and enhancement tasks.
Table 1:
Synthetic datasets: Synthetic datasets (Reddy et al.) [220 [osentino et al.) 2020 Maciejewski eet al. 2020 Chen et al.] (2020b) generate data by convolving Room Impulse Responses (RIRs) with multiple source or noise signals enhance realism, often using image source methods (Allen
& Berkley 1979). While datasets like WHAMR! (Maciejewski et al. 2020) introduce synthetic reverberation and noise, they are created under controlled conditions, limiting their authenticity and inability to fully reflect real-world performance of speech separation and enhancement methods. In contrast, SonicSet ensures acoustic plausibility and offers a wider range of customizable options, such varying geometric structures and multiple microphone array configurations, providing more challenging and realistic environments for these tasks.
Recent advances in reverberation simulation tools have signifi- cantly contributed to the field. FRAM-RIR (Luo & Yu 2023a) uses a stochastic approximation of the image source method for rapid RIR simulation. RIR- Generattt (Tang al. 2020 allows param- eter settings based on the image source method, and Pyroomacoustics (Scheibler et al,, 2018)) offers flexible Python library for fast simulation in 2D 3D rooms. For complex acoustic environments,
Reverberation simulation tools:
3

---

Published as a conference paper at ICLR 2025
COMSOL Multiphysics (Multiphysies 1998) employs finite element methods for detailed room modeling. Compared to these tools, SonicSim based on Habitat-sim (Savva et al., 2019) surpasses others in acoustic realism, producing synthetic audio that closely resembles real- world recordings. also provides practical features for efficiently creating diverse acoustic scenes tailored to specific needs capabilitts lacking in most existing reverberation simulation tools.
MOVING SOUND SOURCE SUITE
The moving sound source suite consists of two components: the customizable data generator SonicSim and the moving sound source dataset SonicSet. SonicSim leverages existing datasets to create data tailored for speech separation and enhancement.
3.1 CUSTOMIZABLE DATA GENERATOR: SONICSIM
The customizable dataset generator, SonicSim, is specifically designed to generate datasets for speech tasks. Built on Habitat-sim (Savva et al. 2019), SonicSim leverages its highly realistic audio renderer (demonstrated in (Chen et al. (022a)) and Sigh-performacc 3D simulator to pro- duce high-quality audio data adapted to various acoustic environments. SonicSim provides a rich set of annotations, including source and microphone position maps, clean audio, and audio with rever- beration and noise, without incurring additional data collection costs. More importantly, SonicSim provides users with extensive control over the dataset generation process, allowing customization of scene layouts, scene materials, source and microphone positions, and microphone types, while ensuring physical realism through its physics engine. By adjusting these configurations, SonieSim can flexibly generate diverse acoustic environment data to meet the specific requirements of var- ious tasks. In Appendix A we discussed in detail the impact of environmental factors on model performance, The following main functions are included in SonicSim, as shown in Figure]
Through Habitat-sim (Savva et al. 2019), SonicSim can import various re- alistic 3D assets generated through simulations or scans, such as those from the Matterport3D dataset (Chang et al. 2017). SonicSim leverages Habitat-sim to interpret metadata and struc- tural information from external datasets and convert them into Habitat-sim' native scene format. Geometric data, material properties, and semantic annotations are transformed to ensure that the imported scenes maintain their original high fidelity. Importing 3D scenes with various shapes, such as cylinders, enables Habitat-sim to simulate different acoustic properties according to dif- ferent spatial layouts. SonicSim allows users to add additional objects to the scene; however, the 3D scenes themselves cannot be modified by users, as they fixed during the import process.
3D scene import:
Acoustic environment simulation: SonicSim uses Habitat-sim to simulate various acoustic fea- tures 3D environments Its capabilities include: 1) accurately simulating sound reflections within room geometries using indoor acoustic modeling and bidirectional path tracing algo- rithms. This effectively simulates the acoustic effect of different objects blocking sound.; 2) mapping the semantic labels of 3D scenes to material properties to set the acoustic features of different surfaces, such as absorption, scattering, and transmission coefficients. SonicSim sup- ports wide range of materials and objects, and users modify the physical and categorical attributes of materials; 3) extended Habitat-sim acoustic simulation capabilities enable the syn- thesis of moving sound source data based on sound source paths.
Microphone types: Habitat-sim offers comprehensive microphone configurations, supporting various audio formats such as mono, binaural, and ambisonic Additionally, we have inte- grated common linear and circular microphone arrays. allowing users to customize the shape of microphone arrays to meet different experimental requirements. Specifically, we first define mul- tiple single-channel receivers corresponding the number of microphones in the array. Next, we configure the spatial relative coordinates of these single-channel receivers based on the desired array shape. Finally, we bind these receivers together and adjust the absolute coordinates to set the position of the microphone array within the environment. We provided a flexible API that enables researchers to represent the desired array layout by inputting custom functions.
attps: http:://hhhaiieee.ooo/o///lessss ssnntttttttiss_..eeeeeeeeeieetse sim, Sim.iiimmmi htw.:mmiiitteooonuattttt..c hemi.
html

---

Published as a conference paper at ICLR 2025
Source and microphone positions: SonicSim allows users to customize randomly set the po- sitions sound sources and microphones, which can be defined within the global coordinate sys- tem of the environment. Apart from static positioning, SonicSim also supports generating motion trajectories for moving sound sources and microphones based on specified start and endpoints. This feature is implemented by interpolating between defined points using navigable paths and leveraging Habitat-sim's 's physics engine to update the positions of moving entities over time. As the sound sources and microphones move along their trajectories, the system dynamically cal- culates the acoustic response in real-time. SonicSim accounts for dynamic changes in distance, occlusion, and environmental interactions, continuously updating the sound propagation paths to simulate evolving reverberation. By integrating these capabilities, SonicSim provides highly realistic simulation environment for scenarios involving moving sound sources.
3.2
MOVING SOUND SOURCE DATASET: SONICSET
(1) Random selection
(2) Random sslection
(1) Generate
(4) positions
(3) Generate moving
postisions
trajectory
Figure 3: An automatic simulation pipeline for moving sound sources. The positions of the sound sources, noises and microphones are all randomly generated. The five RIRs shown in the figure are just for demonstration purposes; the actual process involves more RIRs.
We used SonicSim to construct moving sound source dataset, SonicSet, specifically designed for research on moving speech separation and enhancement tasks. This dataset supports diverse acoustic scenarios by simulating microphones, sound sources, and noise sources randomly placed in the environment. detailed data analysis can be found in Appendix As shown in Figure the entire data generation process is outlined, ensuring the validity of the sound source and microphone configurations under varying positions and dynamic changes.
3.2. DATA SOURCE
The data we used consists of two main components: 3D assets and audio data. For the 3D assets, we utilized the Matterport3D dataset (Chang et al. [017]) a large-scale indoor RGB-D dataset contain- ing 90 building-leve scenes, representing wide range of real-world environments. The semantic segmentation annotations from Matterport3D were used to assign acoustic material properties, en- abling the creation more realistic acoustic environments. In SonicSet, the training set includes 62 scenes, the validation set contains 19 scenes, and the test set includes 9 scenes.
For the audio data, we used speech from the LibriSpeech dataset (Panayotov et al. 2015) and noise from FSD50K (Fonseca et al.) (02T) and FMA (Defferrard et a 2017). LibriSpeech consists of approximately 1,000 hours of 16 kHz read speech from audiobooks; we selected train-clean-33 clean-360 for training. dev-clean for validation, and test-clean for testing. FSD50K. containing 51,197 manually labeled sound events totaling over 100 hours and covering 200 categories, was downsampled to 16 kHz using Librosa (McFee al. 2015) and split into training, validation, and test in 7:2:1 ratio. For FMA, an open music dataset, we used a pre-trained BSRNN model (Luo & Yuj 2023b) remove vocals, ensuring speech separation unaffected during data synthesis. We then downsampled the music to 16 kHz and divided it similarly into training, validation, and test sets.
3.2.2 DATASET CONSTRUCTION
The audio simulation pipeline in the SonicSet dataset, illustrated in Figure 3 consists of the fol- lowing steps. First, we selected a 3D scene from the Matterport3D dataset and imported it into SonicSim to initialize the acoustic environment. Second, we randomly chose positions for the mi- crophone and the sound source within certain layer, with the sound source's 's starting position and the noise source's 's position within a 1-8 meter radius from the microphone. Third, based on the sound source's initial position, we selected endpoint within 1-8 meter range from both the
5

---

Published as a conference paper at ICLR 2025
microphone and the initial position, using SonicSim's trajectory function to generate movement path. Fourth, SonicSim computed the RIRs along the trajectory and convolved them with the source audio. Let s(t) denote the source audio signal of duration T, and let (t) represent the RIRs at discrete positions j = 1, 2.2..... N, where e indexes the audio channels. The convolution at each po- sition is ()()) (t) * ht()). To achieve realistic spatial movement, we interpolated the convolution results of adjacent positions using time-varying interpolation weight (u()) [0. 1]. calculated based on the Euclidean distances between the current position of the moving source r, and the adjacent
positions: (t) dist "ss
dist(ry where dist(ra; rb) denotes the Euclidean distance between positions
Ta and rg. This ensures (t) transitions smoothly from 0 to 1 as the source moves from ry to ry+1-
The interpolated audio signal is then computed as y()) (1 - -a(t)))) ++)(((++(( .y; + (1), dynamic acoustic changes as the source moves (Slaney et al. 1996; Freeland et al., 2002). Finally, interpo- lated audio segments were concatenated form moving source. SonieSim adjusts the source's movement speed based on trajectory length to ensure compatibility with the fixed duration I'.
To create realistic synthetic mixed audio, set each mixed audio clip 60 seconds. Speech seg- ments consisted of 3-5 full audio clips from the same LibriSpeech speaker, arranged into a 60-second sequence with random start positions within 0-8 seconds to ensure varied overlap rates, utilizing SonicSim for moving sound source synthesis. For noise data, we randomly selected 6-8 environ- mental noise segments from FSD50K and 6-8 musical noise segments from FMA, arranging them into 60-second clips with random start positions within 0)4 seconds. Volume levels were adjusted using Loudness Units relative to Full Scale (LUFS): speech at -17 LUFS, environmental noise at -21 LUFS, and musical noise at -24 LUPS. Mixed audio was generated by randomly choosing different speakers and adding environmental or musical noise, with background and musical noise samples including diverse, time-van varying content. In each dataset group (three speech clips plus environ- mental noise clip and one musical noise clip), simulations were conducted within the same scene, simulating speech as moving sources and noise static sources. An example of SonicSet is pre- sented in Figure 5] with corresponding audio metadata and generation hyperparameters provided in
JSON files and in Appendix C
BENCHMARK I: SPEECH SEPARATION
4.1 PROBLEM DEFINITION
Speech separation aims to isolate individual speech signals from a mixture containing multiple speakers, as shown in Figure aa). This task is critical for real-world applications such meet- ings and telephone conversations. A detailed explanation of the task pipeline is provided in Ap- pendix D.I
4.2 BENCHMARK MODELS
We selected several popular models that have demonstrated excellent performance in speech separa- tion as benchmarks, including Conv-TasNet (2019), DPRNN (200)) DPTNet (2020a). SuDORM- RF (2020), A-FRCNN (2021), SKIM (202dd). TDANet (023). BSRNN (0023)) TF-GridNet (202)) Mossformer (2023), and Mossformer2 (2024). For detailed information about the bench- mark models, see Appendix D.2 The PyTorch implementations and pre-trained weights for all benchmark models publicly available
M
xi|
Dicote
Gevenorr
Decoder
(6) Speash soparation
On Speech onhancement
Figure 4: Overall pipeline for speech separation and enhancement.
"The trajectory is generated using Habitat-sim's 's API, speciffclll the "habitat sim. Sim.hhottsstPath method.
httppa //gl www.bccom..cc...cccmmoooo.eateet http://wwwer.tt.te.eeeeeeeeeeeee

---

Published as a conference paper at ICLR 2025
4.3 DATASET DETAILS
We report the performance of trained baseline models using the LRS2-2Mix (Li et al. 2023 Libri2Mix (Cosentino al. 2020), and SonicSet datasets, which we tested on real-world mov- ing speech separation dataset collected by us. We used the test sets from LRS2-2Mix and Libri2Mix to test the models' generalization and transfer performance across different scenarios. Please refer to Appendix D.3 for detailed descriptions of these datasets.
4.4 IMPLEMENTATION DETAILS
To adapt the WSJO-2Mix model (originally for 8 kHz) to our 16 kHz dataset, we doubled the win- dow length and step size of the encoder and decoder, keeping other hyperparameters unchanged. We trained the model using the Adam optimizer (Kingmal 2014) with a batch size of on randomly sampled 3s audio clips, employing negative SNR the loss function. The initial learning rate was set to 1 x 10-3, with gradient clipping and early stopping strategies applied. Detailed hyperparame- ters and training settings are provided in Appendix and
For experiments, we used SonicSet, public datasets, and real-world datasets, with each mixed audio containing two different speakers at 16 kHz. The baseline model was pre-trained on SonicSet using a dynamic augmentation strategy (Luo & Yu 2023b): two speakers are randomly selected from three audio tracks for mixing, adding environmental or music noise as required, and
segment randomly extracted from the 60s au- dio for training. In SonicSet, we used the full test set and processed with a 6s inference window and a 3s sliding window. For the LRS2-2Mis (Li et al. (023) and Libri2Mix (Cosentino et al., 2020) datasets, 2s and 3s audio clips were used for training and testing, respectively.
4.5 EVALUATION METRICS
We employed series of metrics to assess the performance of the speech separation benchmark
models comprehensively, These metrics cover multiple dimensions of audio quality, including signal
quality (SI-SNR (Le Roux et al. 2019) and SDR (Vincent et al. 2006) ). speech intelligibility (STOI
Taal et al., (011) 1 and WER), and subjective quality perception (PESQ (Rix et al. 200T), NISQA
Mittag et al. (021)) and SigMOS (Ristea et al. 2024) ). Details are given in Appendix D.6
4.6 RESULTS AND ANALYSIS
Method
SE-SDR I
SDR
NB-PESQ
WB-PESQ
STOI(56)
MDS Overall
NISQAT
WER
Conv-TasNe
L182.45/3022
3.043.2444822
19922,02212
127.131/1.55
92.3/69.3355.32
2.022.112..25
2,331.261147
847748784474.85
DPRNN
3..54444344
1.32..2
60026447670.33
4765.377.22
91/222
227777
10/.19
S1.10C 18:38117
SaDeRM-R
3.44 79.5.8
26
133.99
X14
09/2.13
4774855
A-FRCNN
87.66002
23
L450
08
21:2.24
86/1.61
4.3447933
TDAN
15/6.11
10:12
5X4054,2943.60
4:3333
39.000
71.2270.4453.84
BSRNN
224/5.0966.15
16659666633
02832.9
71/2.07
2377666
13.22_192.377
18311
S16333
$16333,59446.6
59/49.64
TF-GridNee
6.63 2/111.5
M5-t 13.85
2.002
2622
1.501
Mosstormer
5.727
1011.4
1:10 2..34
4740338438.44
Massormer2
6.668.5511.35
1.50
Table 2: Comparative performance evaluation of models trained on different datasets using RealSEP with environmental noise. The results are reported separately for "trained on LRS2-2Mix", "trained on Libri2Mix" and "trained on SonicSet", distinguished by slash. The relative length is indicated below the value by horizontal bars.
Comparison on real-world datasets. To validate the acoustic simulation gap between synthetic datasets and real data, we constructed a real-world moving sound source dataset consisting of hours of recorded audio (details in Appendix D.3), namely RealSEP, encompassing various complex acoustic environments and different types of background noise (environmental and musical noise). In Tables [ and 3 we trained models using the LRS2-2Mix and Libri2Mix datasets to compare their performance with models trained on the SonicSet dataset. The LRS2-2Mix dataset (Li et al.) 2023) contains real-world noise and reverberation. The Libri2Mix dataset (Cosentino et al. 2020 is currently the largest synthetic speech separation dataset, and its data scale is closest to SonicSet among public speech separation datasets. We trained different models on these datasets and tested them on the RealSEP datasets. addition, we supplemented the experimental results by recording
audio data of direct conversations between individuals (HumanSEP, details in Appendix D.3). We

---

Published as a conference paper at ICLR 2025
Method
SE-SOR
SDR
NB-PRSQ
WI-PESQQ
STOI (56)
MOS Overall
NISQA
WEB
Conv-TasNet
0340488111
0.71
11911261.36
14811880100
11911/201444
97.46996 04
DPRNN
13668.9911.222
188116411-75
13211922.17
DF
310596.18/6611
SaDeRM-RY
10:3333337
1776.32
1022
10/215
0872M1
A-FRCNN
1511
57/1.4
Mo.24h 2779.06
2/22.22
1:20124/1.52
157769045558.44
TDAN
21122072.14
66-32.07
17:2.77
1.280..911.47
SKIM
L81118771.89
1.412.050
BSRNN
98/76
102.3.
13.400955007666
933111.371.00
5.126-2.5244655
TF-GriaNoe
6.81:855770.27
2 72.85
1.501,902.00
T.500
90/2.02
7255m19,7700
0.42.49
18411:00
Mossformer
T016 58.854
5,662,79.994
1:50
555
55.9648.99/98.11
Mosdormer2
1550
5L.449.81/47.33
Table 3: Comparative performance evaluation of models trained different datasets using RealSEP with musical noise. The results are reported separately for "trained on RRxx", "trained on Libri2Mix" and "trained SonicSef", distinguished by slash.
Method
S1-SNR 1
SDR 1
NB-PESQ
WB-FESO
STOI (%)
0v
NISQA
WER (%)
Conv-TasNet
4N114.12
7J3/8.38
2,00/1.84
146/1.42
73.14/33.21
2
1,641.71
DPRNN
17/1.98
163/1.50
7773373
DPTNet
11,11/1166
3,00/12,80
67
2,//2113
$7,62/4.33
7221 L84
PER 332905
SaDoRM-RF
3.01:6.84
9,708 34
2.42/2.15
1.9 1.66
28/2777
2.3522
1 1.491.61
35.6 141.37
A-FRCNN
9.199959
1..63/932
2.702
2.162000
84.32/2.14
1.651.76
35.44:3.82
TDANet
9.2727.00
106088688
2.72/2226
2.32
1.362_19
1.76
30.46/37.16
SKIM
GR276.00
S 7.42
22.23
1675
79.367 7.61
1110:10
1.33:1.46
38.9 242.82
BSRNN
0.86/5666
36
PAUR
TE-GridNet
35.8//4.377
PE 4.43
Mossformer
EE2211
1596/13
3.098
R.39
1669 .98
P4 (V26.64
Mossformer2
14.44/1.12
16.09/12.34
2.83:2.09
91.79/83.24
2.4002:20
1 1.89/2.02
19.51/32.68
Table 4: Comparison of existing speech separation methods on the SonicSet dataset. The per- formance of each model is listed separately for results under "environmental noise" and "musical noise", distinguished by a slash.
then evaluated the NISQA and WER results of the speech separation model trained on different data sets, as shown Table 10 in the Appendix. The results demonstrated that models trained on the SonicSet dataset achieved overall the best separation performance on the RealSEP and HumanSEP datasets. This finding indicates that the SonicSet dataset excels in simulating more realistic acoustic scenes, making it effective for model training and evaluation.
Comparison on the SonieSet dataset. The results of the speech separation are shown in Table Different models exhibit varying performance under environmental and musical noise conditions. Among the RNN-based models, DPRNN and BSRNN performed relatively poorly when handling musical noise. In contrast, CNN-based models (SuDoRM-RF and A-FRCNN) demonstrated more balanced performance across both environmental and musical noise scenarios. Among Transformer- based models, Mossformer2 showed the SOTA performance, especially in handling complex noise. This superiority can be attributed to incorporating the multi-head attention mechanism, which ef- fectively captures long-tem sangttrrm&n dependencies and complex frequency variations (Vaswani al. (017). The detailed discussion is in Appendix D.7
5
BENCHMARK II: SPEECH ENHANCEMENT
5.1 PROBLEM DEFINITION
The speech enhancement task aims to extract high-quality target speech from noisy signal, reduc- ing or eliminating background noise, as shown in Figure (b). This task crucial in applications such speech recognition, speech communication, and hearing aids. A detailed explanation of this task pipeline is available in Appendix E.I
5.2 BENCHMARK MODELS
In speech enhancement experiments, we selected several models: DCCRN (020). Fullband 2021 I),
FullSubNet (202I), Fust-NullSubNett (2022), FullSubNet+ (2022b), TaylorSENet (022a). GaGNet (202c)). G2Net (20226) and Inter-SubNet (202)). The details of these models are provided in
Appendix E.2
8

---

Published as a conference paper at ICLR 2025
5.3 IMPLEMENTATION DETAILS
The hyperparameter settings for all baseline models remain consistent with the original papers. We trained the models using the Adam optimizer (Kingmal 2014) with an initial learning rate of 0.001. If the validation loss does not decrease for five consecutive epochs, we reduce the learning rate by factor of 0.5. All audio samples are processed at a 16 kHz sampling rate. Training employs early stopping and halting if the validation loss doesn't improve for 10 epochs. During testing, we use the same inference strategy as in the speech separation task to handle long audio samples. For (Valentini-
the speech enhancement experiments, we trained models on the VoiceBankDEEMAN Botinhao et al. 2016), DNS Challenge (Reddy et al., 2020), and SenicSet datasets using single- channel speech signals sampled 16 kHz. Baseline models were pre-trained on SonicSet with the same dynamic enhancement strategy as in separation model training, extracting 3s segments from 60s audio for training. The training objective is detailed in Appendix [E.3] For SonicSet, we used the full test set with a 6s inference window and a 3s sliding window to process long audio sequences. On the DoiceBank-DEMMAA and testing, respectively.
and DNS Challenge datasets, 2s and 3s audio clips were used for training
5.4 EVALUATION METRICS
In the speech enhancement task, we use a series of evaluation metrics to comprehensively evaluate the performance of the speech separation baseline model, including signal quality (SI-SNR (Le Roux iet al. (2019) and SDR (Vincent et al.] 2006)), speech intelligibility (STOI (Taal et al. 2011) ) and
CER), and subjective quality perception (PESQ (Rix al. 2001). NISQA (Mittag al. 202
DNSMOS (Reddy et al. (202T) and SigMOS (Ristea et al. 2024)). Since the real-world dataset,
RealMAN is a Chinese dataset, we use CER as the evaluation metric for speech recognition. For
these metrics, except for CER, the higher the value, the better, while for CER, the lower the value, the
better. From an application perspective, CER the most important metric, as speech enhancement usually pre-processing step for speech recognition. The PyTorch implementations and pre-trained weights for all baseline models have been made publicly available
5.5 DATASET DETAILS
Unlike the speech separation task, in the speech enhancement task, we select the audio of a single speaker from the training set as the ground truth label for each iteration, mixing it with various types of noise using the same mixing method as in the speech separation task. We trained baseline models using the training sets of VoiceBank-DEMAND (Valentini- -ecriinn-Bootihho et al.) 2016), the DNS Challenge (Reddy et al. 2020) and SonicSet, and tested them on the test set of the real-world dataset (RealMAN (Yang al. 2024h) to evaluate the effectiveness of different datasets in simulating real acoustic environments. To assess the generalization ability of models trained on SonicSet, we also tested the baseline models VoiceBaan--DEMAND and the DNS Challenge test sets.
5.6 RESULTS AND ANALYSIS
Comparison on real-recorded dataset. We utilized the real-workd recorded moving source dataset, RealMAN (test set), to evaluate trained speech enhancement models on various speech enhancement datasets, including Baiiceaan-DEMMNNU DNS Challenge, and SonicSet. BhiceBank-DEMANN a smaller dataset, while the DNS Challenge dataset comprises approximately 700 hours of data, with data scale comparable that of SonicSet. We employed the DNSMOS (Reddy et al.) 2021) tool to calculate subjective metrics, while Character Error Rate (CER) results were evaluated using the same speech recognition mode??]as as used with ReaLMAN. The results presented in Table 5 indicated that models trained on the SonicSet dataset achieved overall the best results across multiple metrics. Please note that for most practical applications, CER is the most important metric. Moreover, we supplemented the experimental results by constructing dataset called HumanENH (details in Ap- pendix [F.4) by recording human speech audio and noise data. We then evaluated the NISQA and WER results of the speech enhancement models trained on different datasets, as shown in Table T1 in the appendix.
is
ht
waw_ah_ahaa__aaaaaaaaaaaaaaaaaaa char

---

Published as a conference paper at ICLR 2025
Based these findings, we infer that the synthetic dataset SonicSet effectively simulates the acous- tie environments of real moving source scenarios. This provides significant flexibility and cost-
effectiveness for constructing large-scale datasets, positioning synthetic datasets effective al- ternative for addressing complex environments.
Method
SDR
WB-PESQ
MOS Sig
MOS Bak
MOS Overall
NISQA
CER (%)
DCCRN
1,10/187/1.95
L11/1.24/1.26
2.26/3/25/2.44
2.90/2,12/3336
1,90/2.19/2.27
1.84/1.97/1. 97/1666 .62
00.65/37.56/211
Fullband
155/
L04:1:071.27
2.50/2.84/2.55
2.22/2.613.47
2.09/2.99/2446
2.03/2.12/1. 12/19.9
51.67/39.71/20.9
FullSubNet
0.75/1.48/1 1.92
1.101.19/1.33
2.40/.73/2.88
2.94/2.76/3.488
1.94/2.24/2.46
2.14/2.19:1.0 .9
49,23/32.39/19.1
Ne
1.55/1.38/1.37
1.08/11511.31
2.45/3.13/2.67
2.09/2.09/3.48
2.04/1.97/2.59
2.16/2.28:2.08
49.97:40.17/20000
FuttSubNet+
0.58:1664/1.5 1.32
L.1/1.2771.25
2.44/2.51/2.59
2.09/2.87/3.52
2.07/2.31/2.46
2.13/2.17/1.2
45.22/23.
98/20448
TaylorSRNet
1.6/1178/22.66
1.31/3337.331
2.44
68/22.47
2.09/2.63/2.43
2.102222332233
42.55/28.
19/20.6
GaGNet
4.13//L68/2.1 T0
1.071127711
2.627 2.02/2,53/2.46
2.44/3.16/241
2.32/2.35/2.40
1339/1.61/0.77
44.39/34.77/21.00
G2Net
0.01/1,52/2.10
1.101.21/1.229
2.76/2.75/2.445
2.21/2.53/2411
2.07/2.17/2.411
1.41/1.72/0.81
55.12442.98/21.67
Inter-SubNet
1.62:1.551.61
1.091.29/1.34
2.13/.67/2270
3.83/2.88/3447
1.83/2.40/2.51
2.12/2.23/1.94
47.73/22.96/18.75
Table 5: Comparative performance evaluation of models trained on different datasets using the Real-
MAN dataset. The results reported separately for "trained on VoiceBank DEEtankD-EMAD" "trained on DNS Challenge" and "trained on SonicSer", distinguished by a slash. These MOS metrics were calculated using DNSMOS.
Method
SI-SNR
SDR
WB-PESQ
STOI (%)
WER (%)
MOS Noise
MOS Rvrb
MOS Sig
MOS Ovel
NISQA
DCCRN
6,11/1156
2.10/000
2.94..30
3.01/3.51
2.502.94
2.39/2.59
2.01::.96
Fullband
8.3411..0
2.342.02
6226 99
23. 03925.2
3.303.13
3.04/2.99
2.552.86
2.44 +6
2,152 o7
FullSubNett
3,083305
Net
3.14
11.36.36
5.71-11.22
2,08
C00 82
R124999
3.33/3003
1.992.91
2.592
eutt
17.64
1666666
77/999
TastorSENel
10.12
4545
123.46
22.222.76
9222992
65
24
161:1.06
GaGNNe
10.01:12.30
59.5413
21.00233.66
2.71/2.78
86/2.86
2.44/.66
2.23/2.21
T,621.64
9.222..14
12.2113
2./2/8880
S8/2.88
2.44/2.64
2.22/2.22
1.59/6664
Inter-SubNet
11:34:1..0
1287:13.01
2.61/2.28
JK.3/20007
3.39/3.34
3.101
7.62/2 64
1.122115
Table 6: Comparison of speech enhancement methods using the SonicSet test set. The metries are listed separately under "environmental noise" and "musical noise", distinguished by slash. These MOS metries were calculated using SigMOS.
Comparison on the SonicSet dataset. By comparing the performance of different speech enhance- ment models on the SonicSet dataset, we analyzed their enhancement effects under various types of noise. Table 6 presents the performance of the benchmark models. Sub-band splitting methods (such as FullSubNet, Fast-FullSubNet, FullSubNet+, and Inter-SubNet) performed well across mul- tiple metrics. Their interaction processing mechanism between sub-bands significantly improved the models' ability to handle frequency band dependencies, resulting in better speech clarity and noise suppression. In contrast, although TaylorSENet and GaGNet showed relatively similar performance on SI-SNR and SDR, they fell slightly behind in the MOS series scores. The detailed discussion is in Appendix E.5
CONCLUSIONS
We introduce simulation tool named SonicSim, and large-scale synthetic dataset named Sonic- Set, designed to study speech separation and enhancement tasks involving moving sound sources. By integrating the Habitat-sim platform, we developed tool capable of simulating complex acous- tic environments, supporting moving sound sources and multi-scene audio generation. Baseline experiments demonstrate that models pre-trained on the SonieSet dataset exhibit strong generaliza- tion abilities across various public benchmark datasets and real-world recorded datasets, effectively narrowing the gap between simulation and real-world scenarios. Through continuous improvements in the simulation tool and optimization of model algorithms, future work could further advance the deployment of speech tasks in complex environments.
The realism of SonicSim's 's audio simulation is constrained by the level of detail in 3D scene modeling. When there are gaps or incomplete structures the imported 3D scenes, the system cannot accurately simulate the reverberation effects in the current environment.
Limitation:
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research and Development Program of China (No. 20212D0200301) and the National Natural Science Foundation of China (No. U2341228).
REFERENCES
Jont B Allen and David A Berkley. Image method for efficiently simulating small-room acousties. The Journal of the Acoustical Society of America, 65(4):943-955) 1979.
Rohith Aralikatti, Zhenyu Tang, and Dinesh Manocha. Synthetic geoe-geometric impulse responses for improved speech dereverberation. arXiv preprint arXiv:2212.05360, 2022.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niebner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference 3D Vision, pp. 667-676. IEEE, 2017.
Changan Chen, Carl Schissler, Sanchit Garg, Philip Kobernik, Alexander Clegg. Paul Calamia, Dhruv Batra, Philip Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. In Advances in Neural Information Processing Systems, pp. 8896- 8911, 2022a.
Jingjing Chen, Qirong Mao, and Dong Liu. Dual-path transformer network: Direct context-aware modeling for end-to-end monaural speech separation. In Conference of the International Speech Communication Association, 2020a.
Jun Chen, Zilin Wang, Deyi Tuo, Zhiyong Wu, Shiyin Kang, and Helen Meng. Fullsubnet+: Chan- nel attention fullsubnet with complex spectrograms for speech enhancement. In International Conference on Acoustics, Speech and Signal Processing, pp. 7857-7861. IEEE,2022b.
Jun Chen, Wei Rao, Zilin Wang, Jiuxin Lin, Zhiyong Wu, Yannan Wang, Shidong Shang, and Helen Meng. Inter-subnet: Speech enhancement with subband interaction. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 1-5. IEEE, 2023.
Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, Jian Wu, Xiong Xiao, and Jinyu Li. Continuous speech separation: Dataset and analysis. In International Conference on Acoustics, Speech and Signal Processing, pp. 7284-7288. IÉRE, 2020b.
E Colin Cherry. Some experiments on the recognition of speech, with one and with two ears. The Journal of the Acoustical Society of America, 25(5):975-979, 1953.
Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005.11262. 2020.
Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. Fma: dataset for music analysis. In International Society for Music Information Reirieval Conference, 2017.
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra. Fsd50k: an open dataset of human-labeled sound events. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:829 852, 2021.
Fabio P Freeland, Luiz WP Biscainho, and Paulo SR Diniz. Efficient hrtf interpolation in 3d mov- ing sound. Audio Engineering Society Conference: 22nd International Conference: Virtual, Synthetic, and Entertainment Audio. Audio Engineering Society, 2002.
Pierre-Amaury Grumiaux, Srjan Kitic, Laurent Girin, and Alexandre Guerin. A survey of sound source localization with deep learning methods. The Journal of the Acoustical Society of America, 152(1):1071111, 2022.
Xiang Hao and Xiaofei Li. Fast fullsubnet: Accelerate full-band and sub- band fusion model for single-channel speech enhancement. arXiv preprint arXiv:2212.0001 2022.
11

---

Published as a conference paper at ICLR 2025
Xiang Hao, Xiangdong Su, Radu Horaud, and Xiaofei Li. Fullsubnet: A full-band and sub-band fusion model for real-time single-channel speech enhancement. In International Conference on Acoustics, Speech and Signal Processing, pp. 6633-66331 IEEE, 2021.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discrim- inative embeddings for segmentation and separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 31-35. IEEE, 2016.
Xiaolin Hu, Kai Li, Weiyi Zhang. Yi Luo, Jean-Marie Lemercier, and Timo Gerkmann. Speech separation using an asynchronous fully recurrent convolutional neural network. In Advances in Neural Information Processing Systems, volume 34, pp. 22509-22522, 2021.
Yanxin Hu, Yun Liu, Shubo Lv, Mengtao Xing, Shimin Zhang, Yihui Fu, Jian Wu, Bihong Zhang, and Lei Xie. Deern: Deep complex convolution recurrent network for phase-aware speech en- hancement. In Conference of the International Speech Communication Association, 2020.
Daniel P Jarrett, Emanuel AP Habets, Mark RP Thomas, and Patrick A Naylor. Rigid sphere room impulse response simulation: Algorithm and applications. The Journal of the Acoustical Society of America, 132(3): 1462-1472 2012.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint 2014.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey, Sdr balf-eal-baked well done? In International Conference on Acoustics, Speech and Signal Processing, pp. 626-630. IEEE, 2019.
Andong Li, Shan You, Guochen Yu, Chengshi Zheng, and Xiaodong Li. Taylor, can you hear now? a taylor-unfolding framework for monaural speech enhancement. In Proceedings of the
Thirty-First International Joint Conference on Artificial Intelligence, pp. 4193-4200, 2022a.
Andong Li, Chengshi Zheng, Guochen Yu, Juanjuan Cai, and Xiaodong Li. Filtering and refining: A collaborative-style framework for single-channel speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:21 56-2112, 2022b.
Andong Li, Chengshi Zheng, Lu Zhang, and Xiaodong Li. Glance and gaze: A collaborative learn- ing framework for single-channel speech enhancement. Applied Acoustics, 187:108499, 2022c.
Chenda Li, Lei Yang. Weiqin Wang, and Yanmin Qian. Skim: Skipping memory Istm for low- latency real-time continuous speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 681-685. IEEE, 2022d.
Kai Li, Runxuan Yang, and Xiaolin Hu. An efficient encoder-decoder architecture with top-down attention for speech separation. In The Eleventh International Conference Learning Represen- tations, 2023.
PC Loizou. Speech enhancement: theory and practice, 2007.
Francisco Lopez-Hernandes: Rafael Perez-Jimenez, and Asuncion Santamaria. Ray-tracing algo- rithms for fast calculation of the channel impulse response on diffuse ir wireless indoor channels. Optical Engineering, 39(10):2775-277) 27800, 2000.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time- frequenyyy magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, (8)) 1256-1266, 2019.
Yi Luo and Jianwei Yu. Fra-rir: Fast random approximation of the image-source method. In Con- ference of the International Speech Communication Association, 2023a.
Yi Luo and Jianwei Yu. Music source separation with band-split rnn. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1893-1900 2023b.
Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path mn: efficient long sequence modeling for time-domain single-channel speech separation. In International Conference on Acoustics. Speech and Signal Processing, pp. 46-50. IEEE, 2020.
12

---

Published as a conference paper at ICLR 2025
Matthew Maciejewski, Gordon Wicher,, Emnett McQuinn, and Jonathan Le Roux. Whamr!: Noisy and reverberant single-chamnn speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 696-700. IEEE, 2020.
Mishaim Malik, Muhammad Kamran Malik, Khawar Mehmood, and Imran Makhdoom. Automatic speech recognition: survey. Multimedia Tools and Applications, 80:9411-9451 2021.
Brian McFee, Colin Raffel, Dawen Liang. Daniel PW Ellis, Matt Mc Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In SciPy, pp. 18 18-24, 2015.
Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller. Nisqa: deep enn-self- attention model for multidimensional speech quality prediction with crowdsourced datasets. arXiv preprint arXiv: 104. 104.09944 2021.
COMSOL Multiphysics. Introduction to comsol multiphysics@. COMSOL Mulriphysics, Burling- ton, MA, accessed Feb, 9(2018)32, 1998.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In International Conference Acoustics, Speech and Signal Processing, pp. 2206-5210, IEEE, 2015.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference Ma- chine Learning, pp. 28492-28518, 18. PMLR, 2023.
Wei Rao, Yihui Fu, Yanxin Hu, Xin Xu, Yvkai Jv, Jiangyu Han, Zhongjie Jiang, Lei Xie, Yannan Wang, Shinji Watanahe, al. Conferencingspeech challenge: Towards far-field multi-channel speech enhancement for video conferencing. In Automatic Speech Recognition and Understand- ing Workshop, pp. 679-686. IEEE, 2021.
Chandan KA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In Conference of the International Speech Communication Association, 2020.
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objec- tive speech quality metric to evaluate noise suppressors. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6493-6497. IEEE, 2021.
Cicolae-Cätäling Ristea, Ando Saabas, Ross Cutler, Babak Naderi, Sebastian Braun, and Solomiya Branets. Icassp 2024 speech signal improvement challenge. In International Conference on Acousticx, Speech and Signal Processing Workshops, pp. 15-16. IEEE, 2024.
Antony W Rix, John G Beerends, Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment of telephone networks and codecs. In International Conference Acoustics, Speech and Signal Processing, pp. 749-752. IEEE, 2001.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
9339-9349 2019.
Robin Scheibler, Eric Bezzam, and Ivan Dokmanić. Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In International Conference on Acoustics, Speech and Signal Processing, pp. 351-355. IEEE, 2018.
Shilpa Sharma, Punam Rattan, and Anurag Sharma. Recent developments, challenges, and future scope of voice activity detection schemes arrview. Information and Communication Technology for Competitive Strategies (ICTCS 2020) Intelligent Strategies for ICT, pp. 457-464 2021.
Aswin Sivaraman and Minje Kim. Efficient personalized speech enhancement through self- supervised learning. IEEE Journal of Selected Topics in Signal Processing, 144-)1134 2022.
13

---

Published as a conference paper at ICLR 2025
Malcolm Slaney, Michele Covell, and Bud Lassiter. Automatic audio morphing. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, volume 2. pp. 1001-1004. IEEE, 1996.
Alexander Southern, Samuel Siltanen, and Lauri Savioja. Spatial room impulse responses with hybrid modeling method. In Audio Engineering Society Convention 130, Audio Engineering Society, 2011.
Cees H Taal, Richard C Hendriks, Richard Heusdens, and Jesper Jensen. An algorithm for intelligi- bility prediction of time- lime-freeuency weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125-2)) 2236, 2011.
Zhenyu Tang, Lianwu Chen, Bo Wu, Dong Yu, and Dinesh Manocha. Improving reverberant speech training using diffuse acoustic simulation. In International Conference on Acoustics, Speech and Signal Processing, pp. 6969-6973. IEEE, 2020.
Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo rm-rf: Efficient networks for universal audio source separation. In IEEE 30th International Workshop on Machine Learning for Signal Processing, pp. 6. IEEE, 2020.
Cassia -oattinn--otihha, Xin Wang, Shinji Takaki, and Junichi Yamagishi. Investigating rnn based speech enhancement methods for noise-robust text-to-speech. SSW, pp. 146-152, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is you need. Advances in neural information processing systems, volume 30, 2017.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1462- 1469, 2006.
DeLiang Wang and Jitong Chen. Supervised speech separation based on deep learning: An overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):1702-1728 2018.
Qing Hua Wang, Teodor Ivanov, and Parham Aarabi. Acoustic robot navigation using distributed microphone arrays. Information Fusion, 5(2):131-140 2004.
Zhong- Qiu Wang, Samuele Cornell, Shukjae Choi, Younglo Lee, Byeong- Yeol Kim, and Shinji Watanabe. Tf Tf-gridnet: Integrating full-and sub-band modeling for speech separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, al. Chime-6 challenge: Tack- ling multispeaker speech recognition for unsegmented recordings. In International Workshop on Speech Processing in Everyday Environments. ISCA, 2020.
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. In Conference of the International Speech Communication Association, 2019.
Bing Yang, Changsheng Quan, Yabo Wang, Pengyu Wang, Yujie Yang, Ying Fang. Nian Shao, Hui Bu, Xin Xu, and Xiaofei Li. Realman: A real-recorded and annotated microphone array dataset for dynamic speech enhancement and localization. arXiv preprint arXiv:2406. 19959, 2024.
Jun Yu and Chang Wen Chen. From talking head singing head: significant enhancement for more natural human computer interaction. In International Conference on Multimedia and Expo, pp. 511-516. IEEE, 2017.
Shengkui Zhao and Bin Ma. Mossformer: Pushing the performance limit of monaural speech sep- aration using gated single-head transformer with convolution-augmented joint atffaatentions. In International Conference Acoustics, Speech and Signal Processing, pp. 1-5. IEEF, 2023.
14

---

Published as a conference paper at ICLR 2025
Shengkui Zhao, Yukun Ma, Chongjia Ni, Chong Zhang. Hao Wang. Trung Hieu Nguyen, Kun Zhou, Jia Qi Yip, Dianwen Ng, and Bin Ma. Mossformer2: Combining transformer and rnn free recurrent network for enhanced time-domain monaural speech separation. In International Conference on Acoustics, Speech and Signal Processing, pp. 10356-10360. IEFE, 2024.
Victor Zue, Stephanie Seneff, and James Glass. Speech database development at mit: Timit and beyond. Speech Communication, 9(4):351-356, 1990.
A ENVIRONMENTAL FACTOR
We randomly selected 300 audio files from the Librispeech (Panayotov et al.) 2015) test-clean set. 200 of these were used to generate noisy mixtures containing two speakers, and the other 100 were used to generate noisy audio from single speaker. The same audio files were used in both cases. To fix the seene, we chose a cuboid room with no obstructions but included room materials and fixed the positions of the microphone, sound source, and noise. In the experiment occlusion, used Habitat-sim's 's API "add "'bdeooject" to place objects between the microphone and the sound source to simulate the occlusion in sound wave propagation. When studying the effect of room shape, chose cylindrical room from the Matterport3D dataset (Chang et al. 2017) and fixed the relative positions of the microphone, sound source, and noise. In addition, completely removed the root material information from the experiment to explore the effect of the material.
We selected the two methods with the best speech separation and enhancement performance and verified the impact of different environmental factors on the model performance. The experimental results are shown in Tables and The results show that material has the most significant effect on model performance, while the impact of room shape is relatively small. However, all these factors have a certain degree of interference in model performance. Therefore, a simulated dataset using these environmental factors is generated to train the model and improve its generalization ability.
with/without obstacles
cylinder/cuboid
with/without materials
Methods
SDR (t)
WER (%) (L)
SDR (t)
WER (%) (L)
SDR (t)
WER (%) (L)
Mossformer
10.70/11.88 1.84 29.90/24.31
11.18/11.8 1.84
27.17/24.31
11.84/10.12 11.60/9.87
24.31/31.10 25.86/33.58
Mossformer2
10.41/11.60
31.89/25.86
10.78/11.60
30.54/25.86
Table 7: Comparison of the performance of speech separation models for different environmental factors.
with/without obstacles SDR (t)
cylinder/cuboid
with/without materials
Methods
WER (%) (L)
SDR (1)
WER (%) (1)
SDR (1)
WER (%) (1)
FullSubNet
9.67/11 1.05
25.76/19.80
10.56/1 1.05
21.94/19.80
11.05/10.23
19.80/21.1 1.7
Inter-SubNet 10.78/12.69
22.96/17.68
11.50/12.69
19.30/17.68
12.69/1 1.26
17.68/19.88
Table 8: Comparison of the performance of speech enhancement models for different environmental factors.
B DATASET ANALYSIS
The dataset statistics are shown in Table 9 SonicSer dataset offers ground-truth labels for speech separation and enhancement for supervised learning, which captures different audio samples for data synthesis in the same scene, with acoustic environments that closely resemble real-worke conditions.
SonicSet contains 57,596 speech moving trajectories across 90 scenes. The training set includes 102 trajectories, while the validation and test sets each contain 247 trajectories, covering most possible positions within indoor scenes. The dataset comprises 1,001 speakers, with 921 speakers in the training set, 40 in the validation set, and 40 in the test set, ensuring that speaker appears training. Through our data construction
in different subsets, which maintains process, SonicSet generates approximately 952 hours of training data, hours of validation data,
15

---

Published as a conference paper at ICLR 2025
and 4 hours of test data. SonicSet provides researchers with high-quality and complex moving sound source dataset for speech separation and enhancement methods.
Datasets
Speakers
Utterances
Duration (h)
Noise
Reverb
Dynamic
Speech enhancement
TIMIT 1990)
630
6,300
5
x
DoiceBank-DEMATE (0166)
110
44
DNS Challenge (2020)
11k
-100k
760
/
/
x
RealMan 2024)
55
81
/
Speech separation
WSJO 2016
191
28,000
43
X
X
X
EXX
Libri2Mix (2020)
1001
56,800
232
X
XX
LibriCSS 2020b
40
71000
10
V
LRS2-2Mix (2023
100
48,164
50
Speech separation and enhancement
WHAM! 2019
191
28,000
43
X
WHAMR! 2020
191
28,000
43
/
SonicSer (ours)
1001
57,596
960
/
Table 9: Comparison of speech datasets and their characteristics. "Speakers" refer to the number of unique speakers. "Utterances" stands for the total number of utterances, and "Duration" represents the total duration of the dataset in hours. "Noise", "Reverb", and "Dynamic" indicate whether the dataset contains noisy conditions, reverberation, and dynamic acoustic environments, respectively.
C SONICSET DEMO
We provide detailed description of the composition and structure of the SonicSet dataset. Each data group consists of audio files from three mobile sound sources, two types of noise, and trajectory information regarding the movement of these sources. Specifically, each sound source audio file comprises multiple audio segments in FLAC format, with each segment associated with precise start and end times along with corresponding textual content. In addition to speech data, SonicSet includes two categories of interference: background noise and musical noise. The background noise data captures various non-speech sounds in the environment, while the musical noise data consists of music audio recorded within relatively consistent time intervals. Each data group is also accompanied by detailed information sound source trajectories, microphone locations, and noise positions, illustrating the movement patterns of sound sources in different environments. This data format, combining speech, noise, and motion information, provides valuable training data for tasks such as sound source localization (Grumiaax et al. 2022). voice activity detection (Sharma et al.) 2021]. and speech recognition (Malik al. (202). With its rich audio and trajectory information, the SonicSet dataset serves as an ideal testing platform for variety of speech processing tasks, including noisy speech recognition, sound source separation, and source tracking, thereby enhancing the performance of speech and audio-related technologies.
16

---

Published as a conference paper at ICLR 2025
JEFE
3D room map
Movement trajectories
Moving audio
Moving audio
Moving audio III
Environmental noise
Musical nolse
Figure 5: An example from SonicSet data set. Each set of data includes three moving speaker audio files, two types of noise, and the trajectories of the sound source movement.
JSON file 1: The hyperparameters required for audio generation.
"m "mi "microphone":
"type": "monaural",
Opticns: "monaural", "binaural", "Ambisonics"
"Custom array"
"position": {
"x": 0.0,
"y": 1.5,
"2": 0.0
Maudio Qudio_file"; "source_aud wav",
17

---

Published as a conference paper at ICLR 2025
"start :oint":
"x": 5.0,
"y": 1.5,
"2": -3.0
Mend_ :nn_ppoint": {
"x": -2.0,
"y": 1,5,
"2" 4.0
mmovement_tye"" "dynamic", // Options: "static", "dynamic" sudic_settingee "duration": 10.0,
// Duration in seconds @sampling_rater rate": 44100, // In Hz
},
"noise_see seass___cuce" {
=au"i__file";
maccegouud wav"
"positicn";
"x": 2.0,
"y": 1.5,
"z": -1.0
"aucio_settingge"" settings"; { "duration": 10.0, "sampling_rate"; 44100, // In Hz
// curation in seconds
"envircnment":
"scene": "Matterport3D",
JSON file 2:
The raw information corresponding the audio in Figure is stored in file in JSON format. It can be used to train voice activity detection and speech recognition. "ssurcel";
"audio":
*672-122797-0333.f1ea flac",
10.-122797-007..faaa
"672-122197-0045.21 ffac""
"E72-122791-055.flee"
0011:22797-029..laccc
"572-122797-0053.11 flac"
"start_enda Sttart_end_points"::
[83071, 103631],
[118366, 218686],
[360049, 583409]
1632894, 7008941,
1772607, 8214071,
1871263, 918543]
1,
"words": "A STORY",
"THE GOLDEN STAR OF TINSEL WAS STILL ON THE TOP OF THE TREE
AND GLITTERED IN THE SUNSHINE",
"TIME ENOUGH HAD HE TOO FOR HIS REFLECTIONS FOR DAYS AND NIGHTS PASSED ON AND NOBODY CAME UP AND WHEN LAST SOMEBODY DID COME IT WAS ONLY PUT SOME GREAT TRUNKS IN A CORNER OUT OF THE WAY", "I KNOW NO SUCH PLACE SAID THE TREE", "HOW IT WILL SHINE THIS RVENING", "THEY WERE EXTREMELL CURTOUS"
18

---

Published as a conference paper at ICLR 2025
b. "source2";
"audio": [
"908-31957-0014. flac", "908-157963-0007, flac"
"start_eat leiirt__ndd____ points"
191122, 212082], 1268394, 794/14]
words": [
"A RING OF AMETHYST 1 COULD NOT WEAR HERE PLAINER TO MY SIGHT THAN THAT FIRST KISS", "THE LILLY OF THE VALLEY BREATHING IN THE HUMBLE GRASS
ANSWERD THE LOVELY MAID AND SAID I AM WATRY WEED AND AM VERY SMALL AND LOVE TO DWELL IN LOWLY VALES SO WEAK THE GILDED BUTTERFLY SCARCE FERCHES ON MY HEAD YET I AM VISITED FROM HEAVEN AND HE THAT SMILES ON ALL WALKS IN THE VALLEY AND BACH MORN OVER SPREADS HIS HAND SAYING REJOICE THOU HUMBLE GRASS THOU NEW BORN LILY FLOWER"
1
"source3":
"audio":
"61-70968-0021. flac",
"61-70968-0052. flac",
"61-70968-0030. "f
"61-70968-0050- flac" flac"
"61-70970-0006. flac".
"61-70968-0013 flac"
1.
"start_enn__ointss""
[63669, 106709],
[127764, 170164],
[245602, 336562],
[408129, 497409],
[578830, 616190],
1754238, 8254381
1,
"words": "SURELY WE CAN SUBMIT WITH GOOD GRACE", "BUT IS THIS FELLOW PLUCKING AT YOUR SLEEVE",
"NOW BE SILENT ON YOUR LIVES HE BEGAN BUT THE CAPTURED APPRENTICE SET UP AN INSTANT SHOUT" "HE MADE AN EFFORT TO HIDE HIS CONDITION FROK THEM ALL AND ROBIN FELT HIS FINGERS TIGETEN UPON HIS ARM", "NEVER THAT SIR HE HAD SAID", "BEFORE THEM FLED THE STROLLER AND HIS THREE SONS CAPLESS AND TERRIFIED"
},
"noise": {
"audio": : 11, "start end points"; [26850, 940228]
music": {
"audio": [],
"start @itart_end_pae "rtrt__en_poitts:: points:: (13482, 918705]
19

---

Published as a conference paper at ICLR 2025
D BENCHMARK SPEECH SEPARATION
D.1 PROBLEM DEFINITION
Given an input signal = Ec + E RIXT that contains audio z; E RIxT from C speakers and noise n E RIXT, the objective of speech separation to extract each speaker's speech ã IRIxT and assign to different output channels, where 7 denotes the length of audio. Currently, most speech separation methods adopt an encoder-separator-lccode framework (Wang & Chen 2018
In time-domain approaches, the encoder converts y into high-dimensional features E, E IRNx,; through ID convolutional layers, where N and T, denote the feature dimension and length, respec- tively. In time-frequnccy domain approaches, the encoder applies Short-Time Fourier Transform (STFT) to convert y into time-frequency features E E CF where F and T represent the fre- quency and the time dimensions, respectively. Next, E, or Ey is passed to the separator to obtain the feature representations of individual speakers, E, E I x or E E CF Finally, the de- coder reconstructs the target waveform ã; from Et,; or E using transposed convolutional layers or inverse STFT, thereby achieving the final separated speech sources.
D.2 BENCHIAAR MODELS
Cony-TasNet (Luo & Mesgarani 2019): As the first time-domain speech separation model, it em- ploys dilated convolutional network, surpassing traditional frequency-domain methods in sepa- ration performance. This model processes audio directly in the time domain, enhancing clarity compared to frequency-based methods.
(Luo et al. 2020): A time-domain model based on bidirectional LSTM (BLSTM), it in- troduced dual-path architecture for intra- and inter-block modeling, significantly improving the ability to capture long-term dependencies, laying the foundation for subsequent speech separation models. The dual-path approach enhances the model's resolution of speaker characteristics over longer time sequences, optimizing it for complex auditory scenes.
DPRNN
DPTNet (Chen et al. 2020a): Built on DPRNN, it incorporates a multi-head attention mechanism, further enhancing the model's ability to capture long-term contextual information and improving separation performance in complex speech scenarios. As one of the earliest works to introduce the concept of attention into the speech separation, it explores the application of attention in learning complex contextual correlations under the long-term characteristics of the speech domain.
SuDORM-RF (Tzinis et al., 2020); A time-domain speech separation model formed by stacking multiple UNet networks, it uses a refinement strategy to progressively optimize separation perfor- mance, improving the model's ability handle complex signals. This model uses UNet to model different resolutions, and each additional UNet layer refines the separation detail, offering better clarity and fidelity in the final audio output.
A-FRCNN (Hu et al. 2021): Building on SuDORM-RF, it introduces cross-layer connections and parameter-shane sharing mechanisms, improving the model's capacity and parameter efficiency, leading to more precise speech separation. As an asynchronous update scheme, its top-down mechanism enables the model to better model multi-scale temporal information and achieve better results with fewer parameters.
SKiM (Li al. 2022d); Incorporates the reuse of hidden states and cell states in DPRNN's inter- block modeling, strengthening contextual modeling ability and enabling the model better handle long-term dependencies. addition, SKiM also maintains the causal modeling ability the model well, enabling it handle real-time tasks with low latency and good performance.
TDANet (Li et al. 2023): Simplifies redundant connections A-FRCNN and introduces a top- down attention mechanism selectively extract multi-level acoustic features, further improving processing efficiency. The introduction of the attention module further enhances the model's ability to extract complex correlations of temporal information at different scales.
20

---

Published as a conference paper at ICLR 2025
BSRNN (uo & Yu 2023b): Proposes frequency-bande splitting method, modeling both within frequency bands and across the time dimension using BLSTM, marking the first application of band-splitting methods in speech separation. This unique approach allows for more granular control over the frequency components, with better performance tasks that rely more on fine frequency features, such music separation tasks.
TF-GridNet (Wang et al. 2023): Extends DPRNN by modeling in both the time and frequency dimensions and integrates information from both dimensions using multi-head attention module, leading to more refined and comprehensive speech separation. Compared to previous attention- based speech separation works, this work models both time and frequency domains and reduces the length load of the attention module, making it more conducive to capturing contextual relation- ships.
Mossformer (Zhao & Ma 2023): Combines convolution and self-attention in gated, single-head Transformer architecture, utilizing full self-attention within local blocks and linearized, low-cost self-attention mechanism for global modeling, significantly improving separation performance. The model also utilizes an eneetion-gaaheriigg mechanism with simplified single head self-attention. The model achieved good performance by integrating multiple modules.
Mossformer2 (Zhao al., 2024): Enhances Mossformer by introducing a feed-forward sequential memory network (FSMN) recurrent module, improving long-tern dependency modeling and further boosting speech separation effectiivnnsss These FSMN blocks are mainly enhanced by using gated convolution units (GCUs) and dense connections. In addition, bottleneck layers and output layers have been added to control the flow information. Based on the above content, the model integrates the loop module into the MossFormer framework, providing the ability model remote, coarse- scale dependencies and fine-scale loop patterns.
D.3 REAL-WORLD DATASET DETAILS
(2) Office arma large-sia size
Meeting room large-siat lan-ehaped carpet, walls,
Meeting room medium-size prismatic c-ammatic-hhpped carpet,
Meeting room large-size
Meeting room medium-size rectangulaa-ahaaedd corpet,
mectanguler-ehapeed carpel, wooden table, wall
aestangude-ahsped carpet, wooden
Meeding room medium-sbee recangular-hapedd carpet, attract
Meeting room medium-elze eestangula-shaped carpet, sois,
Meeting room small-atze rectanguar-shapedd
Meeting room emali-etre
Office large-ebee
reotangulr-shaped walls
restangular-shee shaped wooden bable, sundries,
Figure 6: Recording audio using realistic scene images. These images show the layout of the actual physical environment used to record audio.
To evaluate the gap between the SonicSet dataset and real-world conditions, well the model's transferability to real-world scenarios, we collected a small-scale real-world moving sound source dataset, namely RealSEP and repeated the relevant experiments. First, we randomly selected 30 audio samples from 10 scenes in the SonicSet validation set, totaling 5 hours of audio. All data was clean, without reverberation or noise. During recording, one participant used the 2023 MacBook Pro's speakers to play audio while moving around randomly within the same scene to obtain the ground-truth audio. addition, we used the same steps to play environmental and music noise from fixed positions to obtain noise data. The audio and noise were recorded using Logitech Blue Yeti Nano omnidirectional microphone fixed in position, with a recording sample rate 16 kHz and 3i-bit depth. After recording, clipped the audio and noise based on the recorded start and stop positions to ensure alignment with the original files. We then recorded another real- wolld dataset in 10 similar real-world scenes using the corresponding original audio. Finally, mixed the data
21

---

Published as a conference paper at ICLR 2025
using the same method as SonicSet construct dataset for testing model performance. The layout of the real-world scenes is illustrated in Figure 6
In addition, have constructed a dialogue speech dataset called HumanSEP, which contains recordings of 16 speakers different scenarios. Specifically, in each scenario, two speakers each randomly read aloud 15 sentences selected from the LibriSpeech dataset (Panayotov et al. 2015), with an average length of 4-6 seconds. To simulate real-world usage scenarios, the speech data was randomly interspersed with environmental noise. All audio was recorded using a fixed-position Logitech Blue Yeti Nano omnidirectional microphone with recording parameters set to a 16 kHz sampling rate and 32-bit depth. The dataset was also designed to include varying degrees of overlap between speakers to increase its diversity and challenge.
To evaluate the generalization performance of the models trained on SonicSet, we used the com- monly used speech separation test datasets LibriMix (Cosentino et al. (0200 and LRS2 (Li et al.) 2023). For the LibriMix dataset, we used the two-speaker mixed test set with 16 kHz sampling rate. For the LRS2 dataset, we used the test set consistent with the (Li et al. 2023)
D.4 TRAINING OBJECT
For all benchmark models in the speech separation, we employed the permutation invariant training (PIT) method (Hershey et al. 2016) to select the best output permutation P Pa (where Pa the set of all M permutations) to minimize the negative SNR adaptively. Specifically, SNR loss LsNR can be defined as a negative SNR:
|
1|
LsnR (Ii; -10 -001og10
|
In this way, the final loss function of the PIT method under the SNR optimization objective can be expressed as:
LPIT.SN- PEРM
min
Σ
LSNR (2,,
D.5 IMPLEMENTATION DETAILS
During training, since all model hyperparameters were originally configured for the WSJO-2Mix dataset (Hershey et al. 2016) and suited for audio with an 8 kHz sampling rate, we doubled the window length and window shift of the encoder and decoder to adapt to the 16 kHz sampling rate datasets. Aside from this adjustment, all other hyperparameters remained unchanged. For training, we randomly sampled 3s audio clips from the training set. The batch size was set to 1, and the Adam optimizer (Kingma 2014) was used with initial learning rate of 1 x 10-3 The learning rate was halved whenever the validation loss did not decrease for five consecutive epochs. We applied gradient clipping to limit the maximum 1.2 norm of the gradients to 5. The training for a maximum of 500 epochs, with early stopping applied if the validation loss did not improve for 10 consccutive epochs.
In the speech separation experiments, we used SonicSet, public datasets, and real-world datasets. Each mixed audio contained two different speakers, with sampling rate of 16 kHz. The baseline models pre-trained on the SonicSet training set were trained using dynamic augmentation strategy. Specifically, during training, we randomly selected set of data, and two speakers were randomly chosen from three available audio tracks for mixing. Depending on the task requirements, either en- vironmental noise or musical noise was then mixed accordingly. Finally, a 38 segment was randomly extracted from the 60s audio for model training.
For each benchmark model, we selected the best model based on its performance on the validation set for testing. In the SonicSet evaluation, we used the full test set and employed 6s inference window with a 3s sliding window to process long audio sequences. For the LRS2-2Mix dataset, used 2s audio segments for both training and testing. For the Libri2Mix dataset, we trained and tested the models using 38 audio segments. All experiments were conducted on a server equipped with 8 NVIDIA 4090 GPUs.
22

---

Published as a conference paper at ICLR 2025
D.6 EVALUATION METRICS' DETAILS
noise Scale-invariant signal-to noise Ratio (SI-SNR) Le Roux et al. 2019) and Signal-to-distone tonalltoo-tstttion ratio (SDR) (Vincent al. 2006) are standard metries for evaluating the ratio of speech noise and dis- tortion, effectively measuring model's ability to suppress noise while preserving the original signal. NB-PESQ and WB-PESQ (Rix al. 200T) are used to assess the perceptual quality of wideband signals, respectively, based on human auditory models, predicting subjective speech quality, which is particularly suitable for evaluating signals processed through separation and enhancement. Addi- tionally, STOI (Taal al. 2011) measures speech intelligibilitty in noisy environments, helping to assess the loss of speech clarity in complex conditions.
Furthermore, Word Error Rate (WER), standard in automatic speech recognition (ASR) systems, measures the error rate between the recognized words and the ground truth, reflecting the intelligi- bility of the speech signal. In this study, use thisper-medium-e- (Radford al. 2023) the ASR model to recognize the content separated audio.
By employing these multi-dimensional evaluation metrics, can comprehensively assess the model's performance in real-world applications.
Method
NISQA
WER (%)
Conv-TasNet
1.37/1.45/1.55
90.61/88. 11/86.97
DPRNN
1.24/1.28/1.31
82.48/766 00/6//666.0/7/299
DPTNet
1.45/1.57/1.63
81.89/74.73/70..
SuDoRM-RF
1.32/1.38/1.4
85.2 85.21/79.31/77
A-FRCNN
1.33/1.40/1.4
77.34/72.44
TDANet
1.31/1.37/1.54
86.61/83.40/7555
SKIM
1.211/.22/1.3
94.43/87.26/85.67
BSRNN
1.54/1.56/1.64
81.92/77.58/73.08
TF-GridNet
1.77/1.83/.96
70.60/64.65/622
Mossformer
L60/1.64/1.76
1.26//..3/7788
Mossformer2
662/1.68/1,78
68.21/62.01/56.558
Table 10: Comparative performance evaluation of models trained on different datasets using the HuamnSEP dataset. The results are reported separately for "trained on LRS2-22ixX, "trained on Libri2Mix" and "trained on SonicSer", distinguished by slash. The relative length is indicated below the value by horizontal bars.
D.7 DISCUSSION OF DIFFERENT METHODS
The performance of different speech separation models varied significantly under environmental noise and music noise conditions, highlighting how architectural choices impact their effectiveness. Among RNN-based models, DPRNN (Luo et al. 2020) and BSRNN (Luo & Yu 2023b) performed relatively poorly when handling music noise. This suggests that these models face limitations in processing complex background music, likely due to the frequency variations in music noise posing significant challenges for LANN-based architectures. While recurrent architectures like DPRNN and BSRNN are effective in modeling long-term dependencies through bidirectional LSTM networks, they may not capture intricate contextual relationships as efficiently, especially in the presence of complex noise sources like music.
In contrast, CNN-based models such as SuDoRM-RF (Tzinis et al. 2020), A-FRCNN (Hu et al. 2021), and TDANet et al. 2023) demonstrated more balanced performance across environmen- tal and music noise scenarios. Their advantages in spectral refinement and signal reconstruction, particularly when leveraging multiple U-Net blocks, enabled them to effectively address complex noise interference. The incorporation of cross- cross-layer connections and in TDANet enhances parameter efficiency and increases the model's capacity without proportionally
mechanisms
23

---

Published as a conference paper at ICLR 2025
increasing computational load, leading to more precise speech separation. This design choice strikes a balance between model complexity, parameter efficiency, and performance, making TDANet more suitable for real-world applications where computational resources are constrained.
Among Transformer- based models, DPTNet (Chen et al. 2020a), TF-GridNet (Wang et al. 2023), Mossformer (Zhao & Ma 2023). and Mossformer2 (Zhao al. 2024) showed the best overall performance, especially in handling complex music noise while maintaining high separation accu- racy. This superiority can be attributed the multi-head attention mechanism, which effectively captures long-range dependencies and complex frequency variations. Attention mechanisms en- hance the ability to model long-range dependencies and complex contextual correlations within speech signals, contrasting with the limitations of recurrent architectures. In music noise scenarios, this mechanism leverages contextual information to significantly reduce signal distortion, producing separated speech that is both clear and intelligible.
Furthermore, models like TF-GridNet (Wang et al. 2023) and Mossformer (Zhao & Ma 2023) not only excelled in SI-SNR and SDR scores but also demonstrated high intelligibility scores, indicating their proficiency in producing separated speech that maintains both clarity and naturalness. TF- GridNet's innovative approach extends the dual-path recurrent neural network by modeling in both time and frequency dimensions and integrating information through multi-head attention module. modeling strategy effectively captures temporal and spectral information, leading to significant improvements in separation performance.
This
The trade-off between model complexity, parameter efficiency, and performance is a critical aspect highlighted in the analysis. While top-performing models with advanced attention mechanisms offer superior separation quality, their computational complexity may pose challenges for real-time deployment. Latency and resource requirements become crucial factors when implementing these models in live systems. Therefore, models like A-FRCNN, which optimize computational efficiency without compromising performance, may be more practical for applications where resources are limited.
E BENCHMARK II: SPEECH ENHANCEMENT
E. PROBLEM DEFINITION
Typically, the speech enhancement problem can be expressed as follows: given an observed noisy speech signal s E R IxT, where s x + n, with æ E I being the target speech signal and n E IR 1x? representing background noise, the goal of speech enhancement is to recover an estimate a of the target speech signal from s. Specifically, the process begins by applying the STFT to map the time-domain signal s into time-frequency domain features E CFXT", Then, a speech enhancement network extracts the target speech from the noisy mixed signal, producing estimated time-frequent domain features = fEE). Finally, the Inverse STFT is applied to reconstruct E E XX" back into the time-domain waveform & E I 1xT thus completing the recovery of the target speech signal.
E.2 BENCHMARK MODELS
DCCRN (Hu al. 2020 is speech enhancement model designed for complex-domain processing. It operates in the frequency domain by handling both the real and imaginary components of speech signals, leveraging the characteristics of complex signals. Additionally, DCCRN employs a recur- rent neural network to capture temporal information, making it particularly effective in processing speech in complex noisy environments. The model's innovation lies in its combination of complex- domain processing and temporal feature extraction, providing robust support for improving speech clarity and intelligibility.
Fullband (Hao et al., 2021) processes the entire frequency band of the speech signal, typically operating directly on either the full spectrum the time or frequency domain. These approaches particularly well-suited for speech enhancement tasks that address full-band noise. Leveraging complete frequency information effectively improves speech intelligibility and quality, making them strong choice for applications in diverse noisy environments. Their comprehensive processing
24

---

Published as a conference paper at ICLR 2025
capability ensures that both spectral and temporal features are preserved, leading more natural and clear enhanced speech outputs.
FullSubNet (Hao et al. 2021) is speech enhancement model that effectively combines full full-band and sub-band information. It begins by extracting global features at the full-band level, subsequently refining these features the sub-band level. This dual-layer processing approach enhances the over- all performance of speech enhancement, allowing for more precise noise reduction and improved clarity. By integrating information from both frequency scales, FullSubNet successfully addresses various noise conditions, leading to more intelligible and natural-sounding speech outputs.
Fast- Funt-FullSubNet (Hao & Li 2022) is an accelerated version of FullSubNet, specifically optimized for real-time speech enhancement. This model reduces computational complexity and improves in- ference speed, allowing faster speech processing while maintaining high enhancement quality. By streamlining the architecture and leveraging efficient algorithms, Past-FullSubNet addresses the de- mands of real-time applications, making it suitable for scenarios where quick responses are critical.
FullSubNett (Chen et al., 2022b) is an improved version of FullSubNet that further optimizes the model architecture and parameters for more efficient handling of sub-band information. incorpo- rates advanced techniques, such as channel attention mechanisms, to boost performance, particularly in complex noise environments. By refining the processing strategies and utilizing complex spectro- grams, FullSubNet+ achieves superior noise reduction and clarity compared to its predecessor.
TaylorSENet (Li et al., 2022a) is a speech enhancement model that utilizes the Taylor series ex- pansion to represent speech signal features. By expanding these features
Taylor series, the model captures subtle variations in the signal, which enhances its robustness when processing com- plex audio signals. This innovative approach allows TaylorSENet improve the overall quality and intelligibility of speech in challenging acoustic environments.
GaGNet et. 2022c) introduces novel approach that combines gating mechanisms and guided learning to enhance speech separation tasks. The gating mechanism plays a crucial role in control- ling the flow of information within the model, allowing for more effective differentiation between target speech and background noise. Simultaneously, the guided learning mechanism leverages external prior knowledge to assist the model in improving its separation capabilities. This dual ap- proach not only enhances the model's performance in challenging acoustic environments but also facilitates better intelligibility and clarity of the separated speech, making GaGNet a significant advancement in the field of speech processing.
G2Net (Li et al. 2022b) enhances computational efficiency and performance by incorporating both gating mechanisms and group convolution. The use of group convolution effectively reduces the number of parameters in the model, leading to lower computational costs. Meanwhile, the gating mechanism plays critical role in helping the model selectively process and prioritize key speech features. This combination not only optimizes the performance model but also makes it suitable for real-time speech enhancement tasks, ensuring that it delivers high-quality outputs even in demanding acoustic environments.
Inter-SubNet (Chen al. 2023) is speech enhancement model that focuses interaction pro- cessing among sun-bands. By decomposing sub-band features, it facilitates information exchange between different sub-bands, effectively addressing inter-band dependencies. This innovative ap- proach enhances the model's 's ability to capture the complex relationships between sub-bands, lead- ing improved speech enhancement outcomes. Through this interaction, Inter-SubNet significantly boosts the quality and intelligibility of the enhanced speech, making it particularly effective in chal- lenging acoustic environments where traditional methods may struggle.
E.3 TRAINING OBJECT
For all benchmark models used in the speech enhancement tasks, we applied various objective func- tions as outlined in the original papers to optimize model performance. Specifically, the SI-SNR loss
function was employed in the DCCRN model (Hu et al.] 2020). For the Fullband Hao et al. 2021
FullSubNet Hao et al. (202)). FullSubNet+ (Chnn Chen et al. (2022b), FullSubNet- Fast Hao & Li and Inter-SubNet Chen et al. (2023) models, utilized the complex ratio mask (cRM) error in the frequency domain as the core loss metric. By computing the mean square error (MSE) between the accurately quantify the model's
2022
complex mask of the enhanced speech and the ideal mask,
25

---

Published as a conference paper at ICLR 2025
performance in the time- time-frequency domain. speech signal but also effectively preserves its phase information. The loss function expressed as:
This method not only restores the magnitude of the
Fullland = MSE(M RRMM M aIRM)
(3)
where M MRRM E CFXT" is the estimated complex ratio mask, and MRRM E CFxT is the ideal complex ratio mask.
For the TaylorSENet (Li et al. 2022a) and GaGNet (Li et al. 2022c) models, we used an Euclidean loss function based on complex magnitude. This loss function not only considers the differences in the complex space but also incorporates magnitude information, enabling the model to capture subtle changes in the input speech signal more precisely. The objective function is given as:
TaaykosSNNe a Le (1 ox) Lms
where is the weighting factor that controls the trade-off between the complex error Le and the magnitude error Lm- Le is the loss of the complex part, which is defined as:
Le x)) - T xr! [
x
(5)
where 2, and 2; represent the real and imaginary parts of the predicted STFT, respectively, and ø and z; are the real and imaginary parts of the ground-truth STFT. L.n is the loss of the amplitude part, which is defined as:
L (ā) ((z) |x|)²
In G2Net (Li et al., 2022b), we employed the stagewise complex magnitude Euclidean loss. This loss function applies different weights to the estimated signals at various stages, allowing the model to converge quickly in the early stages while fine-tuning the output signal quality the later stages.
The formula is follows:
LGNNet Σ a; -G- (L, + L.n x)),
where a; is the weight for each stage, and N is the number of stages in the model.
E.4 REAL-WORLD DATASET DETAILS
To evaluate the gap between the SonicSet dataset and real-world conditions, and the model's trans- ferability to real-world scenarios, we collected small real-world speech dataset for speech en- hancement called HumanENH, which contains recordings from 16 speakers covering scenarios. Each speaker read 10 sentences from the LibriSpeech dataset (Panayotov et al.) 2015) in the specified scenarios, with each sentence lasting 4-6s. During recording, we randomly added different ambient noises, such as traffic noise, cafe background noise and office ambient noise, to simulate real-world usage scenarios. All audio was recorded using a fixed-position Logitech Blue Yeti Nano omnidirectional microphone with recording parameters set to a 16 kHz sampling rate and 32-bi bit depth. The dataset is designed to be diverse and challenging, facilitating the evaluation of model performance speech enhancement tasks.
different
E.5 DISCUSSION OF DIFFERENT METHODS
The performance variations among the different speech enhancement models can be attributed to their underlying architectures and processing strategies. Subband-splitingg methods like FullSub- Net (Hao et al. 2021), Fast- Fast-FullSubNet (Hao & Li 2022). FullSubNet+ (Chen et al.) 2022b), and Inter-SubNet (Chen et al. 2023) excelled across multiple evaluation metrics due to their ability to handle frequency band dependencies effectively. By splitting the speech signal into subbands and facilitating inter-subband interaction, these models can capture detailed frequency-specific features and relationships. This approach enhances their capability to suppress noise and improve speech clarity, it allows the models address both local and global spectral variations more precisely. The interaction between subbands enables better modeling of complex acoustic patterns, leading to superior performance in terms of speech intelligibility and quality.
26

---

Published as a conference paper at ICLR 2025
Method
NISQA
WER (%)
DCCRN
2.01/2.08/1.59
12.91/62..9/52.16
Fullband
2.12/2.19/1.78
46.98/36.48/299888
FullSubNet
2.23/2.31/1.81
69.64/55.71/47.13
Fast-FullSubNet
2.24/2.29/1.81
54.87/42..4/333288
FullSubNet+
2.33/2.39/1.79
70.57/62.52/55.23
TaylorSENet
2.66/2.68/1.4b
38.70/30.12/24.13
GaGNet
2.57/2.61/1.544
39.88/33.65/24.94
G2Net
2.42/2.42/1.49
45.92/40.28/30.94
Inter-SubNet
1.76/1.87/1.84
63.36/54.53/42.07
Table 11: Comparative performance evaluation of models trained on different datasets using the HumanENH dataset. The results are reported separately for "trained on VoiceBank-DEMANDD" "trained on DNS Challenge" and "trained on SonicSef", distinguished by slash.
In contrast, models like TaylorSENet (Li et al. 2022a) and GaGNet (Li et al. 2022c demonstrated strong performance in objective metries related to noise reduction, such SI SI-SNR and SDR, indi- cating their effectiveness in suppressing background noise. However, they slightly lagged in subjec- tive metrics like MOS, which assess perceived speech quality, including aspects like reverberation and signal fidelity. The possible reason for this disparity is that while these models are adept at denoising, they might not preserve the naturalness of the speech signal as effectively. Their archi- tectures may not fully capture the nuances of reverberation effects or may introduce artifacts that affect the overall listening experience, suggesting limitations in maintaining signal integrity during the enhancement process.
27


Reviews:
[{"review_id": "WGbNKCGyES", "reviewer": "Reviewer_h9Ga", "summary": "The paper proposes a system and dataset for stimulating moving source sound data. The system is built using Habitat-sim which takes in a 3D scene and stimulates audio for receivers and transmitters placed at different locations. Simulated data for speech separation and speech enhancement problems are created, which are then used to benchmark performances of several state-of-the-art models for these problems. The results show that training models on simulated data can lead to better results on real-world dynamic audio.", "questions": "Please see the questions above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "– Simulating moving sound sources is indeed challenging and there are many good solutions for this problem. The paper relies on Habitat-sim and is a good attempt at improving simulation of moving sound sources. \n\n– From audio perspective, the proposed system provides a reasonable degree of freedom to simulate in various conditions.  \n\n– The benchmarking of different speech separation and enhancement models is good – a variety of models for both tasks.", "weaknesses": "– The details of the moving source simulation could be improved a lot and is a major concern in the paper. Is the paper following  some known method ? Some mathematical description of what is going on in the moving source case is desirable, especially for description in Section 3.2.2 ? What exactly is SonicSim trajectory function “We then employed SonicSim’s trajectory function to calculate and generate a movement path for the sound source ” ? \n\n– Is the speed of the moving source a factor in the whole simulation process ? \n\n– Some ground truth validation of the simulations maybe helpful. For example, record data for a moving source in a real room and then use the 3D scene input of that room to simulate some data. How close are they ? \n\n– While the benchmarking on speech enhancement and separation are nice, a more useful benchmarking might be RIR estimation ? There are plenty of recent works on neural room acoustics generation and I think this problem would provide more insightful picture of the simulation system. \n\n– the paper mentions several challenges in audio simulations like obstacles, room geometry, room surface but none of them end up being a factor studied by the paper. \n\n– some aspects of the audio generation settings like the microphone type, monoaural vs binaural might be good to showcase.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s5FkRRDdHd", "reviewer": "Reviewer_ADhE", "summary": "SonicSim provides a new data simulator for moving sound sources for researchers working on speech separation and speech enhancement.  SonicSet is constructed using SonicSim using LibriSpeech, FreesoundDataset50k, and FreeMusicArchive, and 90 scenes from Matterport3D.\n\nUsing 9 speech separation models the authors show improvements when training on Voice-DEMAND, DNS Challenge, and SonicSet, and tested on the RealMAN test set, which shows significant improvement using SonicSet.", "questions": "What is the 5-hour dataset described in C.5 called (please give it a name - it is not SonicSet). Also why was a MacBook Pro used to play back speech? That seems like a very poor choice to simulate a mouth. Why not use an artificial mouth instead, which mimics the response of a human speaker (frequency response and directionality)?\n\nHow can new microphone types be added?\n\nTable 5: Says DNSMOS is used but it looks like SIGMOS used also? Please clarify which model MOS Sig, MOS BAK, and MOS Overall come from. \n\nTable 6: Why are SI-SNR and WB-PESQ results much lower for Inter-SubNet than in the Inter-SubNet paper?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "For researchers doing speech separation and enhancement with moving sound sources SonicSim and SonicSet will be valuable tools to create training data and training models. It fills an important gap in speech separation and enhancement dataset simulation.", "weaknesses": "SonicSim is constrained by the level of detail in 3D scene modeling. \n\nFor evaluation a subjective rating could have been used as done in DNS Challenge 2023 which would be more convincing than the objective metrics used (other than WER which is excellent).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7wRzTaK994", "reviewer": "Reviewer_jUje", "summary": "The paper introduces SonicSim, a customizable simulation platform designed to generate synthetic data for moving sound source scenarios, addressing limitations in real-world and existing synthetic datasets for speech separation and enhancement models. Built on Habitat-sim, SonicSim enables multi-level adjustments across scene, microphone, and source levels, providing flexibility for diverse scenarios. Using SonicSim, the authors created SonicSet, a benchmark dataset derived from Librispeech, FSD50K, FMA, and Matterport3D to evaluate models under moving sound conditions. Additionally, real-world data comparisons validate SonicSim's synthetic data, showing promising generalization to real-world scenarios.", "questions": "1. Have the authors considered implementing advanced full-reference, as well as no-reference MOS metrics such as NISQA or SQAPP, or even non-matching reference metrics? These could enhance the quality assessment of synthetic data and offer more insight into the real-synthetic data gap.\n\n2. How well does SonicSim perform in dynamically noisy environments, where background sounds change over time? Would it be feasible to test in such conditions to increase practical applicability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers a toolkit for generating customizable synthetic data for speech processing in moving sound source scenarios, an area where realistic data is scarce. SonicSim stands out for its versatility in multi-level adjustments and the breadth of scenarios it enables. The SonicSet dataset, derived from a combination of diverse sources, is a method for benchmarking models in this field. The authors also make an effort to bridge the synthetic-real data gap by comparing SonicSet's synthetic data with real-world datasets, which enhances the reliability of SonicSim as a training and evaluation tool. Evaluations against evaluation metrics are pretty solid.", "weaknesses": "One potential limitation is the lack of human evaluation MOS metrics for quality measurement. Existing metrics like PESQ and DNSMOS are used, but the paper lacks inclusion of more advanced full-reference, as well as no-reference models like NISQA or SQAPP. These metrics could provide a statistically significant, nuanced understanding of synthetic data quality versus similarity to real-world audio, strengthening the claims around SonicSim's effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6xx67UrdK", "reviewer": "Reviewer_xZx6", "summary": "This paper presents SonicSim, a toolkit for generating realistic synthetic data in dynamic audio environments, specifically for moving sound sources. Built on Habitat-sim, this platform allows users to control variables like scene configurations, microphone types, and source movements. Using SonicSim, the authors created SonicSet, a new dataset to test models for speech separation and enhancement in several conditions. Results suggest that models trained on SonicSet perform better in real-world settings than those trained on other synthetic datasets.", "questions": "See weaknesses. I'd be happy to increase my rating if the authors address the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. SonicSim provides a customizable environment where users can adjust elements like scene layouts, source paths, and microphone setups.\n2. SonicSim and SonicSet offer a promising step toward bridging the gap between synthetic and real-world audio data, which addresses current limitations in data for training speech front-end models.\n3. The paper presents well-organized experiments, testing a variety of speech separation and enhancement models with a range of evaluation metrics like SI-SNR, SDR, and WER. These thorough experiments help demonstrate the value of SonicSet in preparing models for real-world conditions.\n4. I appreciate the authors’ effort to make their work reproducible by providing open-source code and dataset.", "weaknesses": "1. I find the real-world testing too limited, as it relies heavily on the RealMAN dataset for speech enhancement (Section 5.5). Expanding testing to include other real-world data, perhaps by offering qualitative results on some in-the-wild examples (where speech is from actual conversations rather than recorded playback), would give a more comprehensive picture of how models trained on SonicSet generalize. That said, RealMAN alone may not be enough to fully support the claims of SonicSet’s real-world effectiveness, and an additional perceptual study would be highly beneficial.\n2. Although the authors claim that SonicSim improves realism in reverberation modeling, there isn’t enough quantitative or qualitative support for this. I would recommend directly comparing SonicSim’s synthetic RIRs to real RIRs to demonstrate this realism. This type of comparison could show how much the enhanced realism of SonicSim truly benefits model performance.\n3. While the paper covers a range of models for speech separation and enhancement, it’s unclear why these specific models were chosen. I think the authors should consider discussing a bit why one model is better than the other, and including more analysis to clarify how well SonicSim supports different model types.\n4. SonicSet uses fixed LUFS levels for environmental and musical noise, which may limit how robust models are to different real-world noise levels. Adding tests with varied noise levels (like signal-to-noise ratios (SNRs)) could help demonstrate SonicSet’s ability to handle diverse noise conditions. I would also be interested to see how varying the noise characteristics affects model performance.\n5. It’s not clear to me how flexible SonicSim is regarding scene layouts and material properties. It would help if the authors explained what can be adjusted versus what is fixed. This would allow readers to understand any limits in SonicSim’s adaptability.\n6. Moving key details (training details, dataset splits, and evaluation metrics) into the main text would improve readability and reduce the need for readers to frequently switch sections, especially for understanding the experimental setup. The current Introduction section feels overly detailed and could be streamlined (reducing adverb use would improve clarity).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)