<figure><img src="image_1.png" /></figure>

HAL
open science

# Physically Informed Spatial Regularization for Sound Event Localization and Detection

Haocheng Liu, Diego Di Carlo, Aditya Arie Nugraha, Kazuyoshi Yoshii, Gaël Richard, Mathieu Fontaine

<figure><img src="image_2.png" /></figure>

## To cite this version:

Haocheng Liu, Diego Di Carlo, Aditya Arie Nugraha, Kazuyoshi Yoshii, Gaël Richard, et al., Physically Informed Spatial Regularization for Sound Event Localization and Detection. IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, Oct 2025, Tahoe City, California, United States. hal-05244860

HAL Id: hal-05244860

https://hal.science/hal-05244860v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Physically Informed Spatial Regularization
for Sound Event Localization and Detection

Haocheng Liu¹, Diego Di Carlo², Aditya Arie Nugraha², Kazuyoshi Yoshii²,³, Gaël Richard¹, Mathieu Fontaine¹

¹ LTCI, Télécom Paris, Institut Polytechnique de Paris, Palaiseau, France

²AIP, RIKEN, Tokyo, Japan ³Kyoto University, BFE Lab, Japan

***Abstract***—Building Sound Event Localization and Detection (SELD) models that are robust to diverse acoustic environments remains one of the major challenges in multichannel signal processing, as reflections and reverberation can significantly confuse both the source direction and event detection. Introducing priors such as microphone geometry or room impulse response (RIR) into the model has proven effective in addressing this issue. Existing methods typically incorporate such priors in a deterministic way, often through data augmentation to enlarge data diversity. However, the uncertainty arising from the complex nature of audio acoustics remains largely underexplored in the SELD literature and naturally call for incorporating a stochastic modeling of acoustic prior. In this paper, we propose regularizing deep learning based SELD models with a physically constructed spatial covariance matrix (SCM) based on the estimated direction of arrival (DOA) and sound event detection (SED).

# 1. INTRODUCTION

Sound event localization and detection (SELD) involves determining the temporal occurrences of specific sound classes and their associated directions of arrival (DOA) using audio signals captured by one or several microphones. This task is highly relevant in areas such as machine listening [1], anomalous detection [2], or autonomous surveillance [3], and is also emerging in edge computing applications on mobile devices [4].

Deep neural networks (DNNs) have demonstrated impressive results in SELD tasks. Several studies have explored end-to-end approaches that directly leverage multichannel raw waveform input, allowing convolutional networks to learn feature representations [5]. However, the main methodologies have predominantly favored explicit time-frequency transformation preprocessing, focusing on well-established acoustic feature extraction techniques to provide robust model input which combines multichannel time-frequency spectrum and explainable spatial feature [6]–[8].

In addition to the magnitude and phase difference of the spectrum, the spatial covariance matrix (SCM) effectively captures the structural characteristics of microphone array observation, and is widely used in multichannel audio data modeling. It is commonly used in a various spatial acoustics tasks, including source localization [9], source separation [10], [11], beamforming [12], and sound field analysis [13], due to its ability to represent spatial correlations and directional properties of sound sources. In machine learning-based approaches to these tasks, the SCM is also leveraged for constructing loss functions [14]–[16]. In addition, it implicitly encompasses crucial acoustic information such as the room impulse response (RIR) and allow for formalising a stochastic model of the audio scene [17].

RIRs characterizes how sound propagates within an acoustic environment, representing both direct sound paths and reflections. This makes RIR-based methods particularly valuable for acoustic modeling in reverberant environments. RIR-SF [18] constructed spatial fea-

<figure><img src="image_4.png" /><figcaption>Fig. 1: Overview of the proposed framework.</figcaption></figure>

tures with RIR in multichannel speech recognition. Reverberation and early reflections are also modeled as a feature in distance estimation-related tasks [19].

Most SELD methods use phase-based spectrograms and class and directional labels, but overlook the alignment with real microphone geometry and room acoustics [6], [7], [20]. In [21], the sound pressure level (SPL) and the acoustic attenuation prior are modeled in SELD for DOA and distance estimation. However, their model needs to know the event to estimate these parameters. In parallel, geometrical acoustic simulations and measured RIR were used in data augmentation to improve the performance of the SELD system [22], [23], but directly and physically integrating the acoustics prior into the modeling and training framework remains an underexplored area [24]. To take into account the acoustics in deep learning-based SELD models, we propose a regularization-based framework (Fig. 1) that introduces a physically informed prior in the training process. This prior bridges the input observations and output representations by enforcing consistency with the room acoustics and microphone array structure, and encouraging the model to produce physically plausible predictions. Our contribution are as follows.

(i) We propose a physically informed inverse Wishart regularization applied to the SCM to enhance deep learning-based SELD models.

(ii) We investigate and evaluate two theoretical approaches for SCM construction: an algebraic steering vector-based method and a RIR retrieval-based method.

(iii) We conduct experiments under various acoustic conditions and data regimes, demonstrating the superiority of the proposed regularization approach.

To summarize, our regularized method implies using a physically coherent loss added to a backbone SELD system without increasing the inference complexity nor the parameter number. The results overall show that the proposed regularization not only improves overall model performance, but also provides strong robustness under limited-data training, with less performance drop compared to the baseline model.

This work was supported by ANR Project SAROUMANE (ANR-22-CE23-0011) and was partially funded by Hi! PARIS and ANR/France 2030 (ANR-23-IACL-0005). It also received support from Project JST FOREST no. JP-MJFR2270, JSPS KAKENHI nos. JP23K16912, JP23K16913, 24H00742, 24H00748, and 25H01142.
## 2. BACKGROUND

Given an M-channel complex short-time Fourier transform (STFT) spectrogram $\mathbf{X} \triangleq \{\mathbf{x}_{ft}\}_{f,t=1}^{F,T} \in \mathbb{C}^{F \times T \times M}$, where $\mathbf{x}_{ft} \triangleq [x_{1,ft}, \dots, x_{M,ft}]^\top \in \mathbb{C}^M$, the sound event localization and detection (SELD) methods extract time-frequency (TF) features denoted by $\mathbf{\Phi}(\mathbf{X})$ to estimate the sound event classes $\hat{\mathbf{Y}}_{\text{SED}}$ and the sound event directions $\hat{\mathbf{Y}}_{\text{DOA}}$. In practice, deep neural networks (DNNs) typically have two parts, i.e., a multi-label classification part for $\hat{\mathbf{Y}}_{\text{SED}}$ and a regression part for $\hat{\mathbf{Y}}_{\text{DOA}}$ that predicts 3D DOA on the unit sphere. We briefly describe the widely-use representations for the input $\mathbf{\Phi}(\mathbf{X})$ in Sec. 2.1 and those for the outputs $\hat{\mathbf{Y}}_{\text{SED}}$ and $\hat{\mathbf{Y}}_{\text{DOA}}$ in Sec. 2.2.

### 2.1. Input features

Many SELD models take as input a stack of multiple features (representations) extracted from $\mathbf{X}$, e.g., the scaled magnitude spectrogram and the following phase-aware features.

**Eigenvector-based phase vector (EPV)** approximates the relative distances of arrival (RDOAs) and is computed using

$$ \mathbf{V}_{2:M,ft} = -c(2\pi f)^{-1} \arg[u_{1,ft}^* \mathbf{U}_{2:M,ft}], \quad (1) $$

where * denotes the complex conjugate, c represents the speed of sound, $\mathbf{U}_{l:l',ft} \in \mathbb{C}^{(l'-l+1) \times M}$ are the $l' - l + 1$ eigenvectors associated to the $\{l' \dots l'\}$ eigenvalue decreasingly sorted of the observed covariance matrix

$$ \tilde{\mathbf{G}}_{ft} \triangleq \mathbb{E}[\mathbf{x}_{ft} \mathbf{x}_{ft}^\mathrm{H}] \in \mathbb{C}^{M \times M}, \quad (2) $$

and $(\cdot)^\mathrm{H}$ means the Hermitian transpose and $\arg(w\mathbf{Z})$ means the argument between the scalar $w$ and all the entries of a matrix or vector $\mathbf{Z}$. The eigenvector mitigates the impact of spatial noise, resulting in cleaner representations for downstream network learning and processing. By stacking the EPV with linear-frequency log-power spectrograms, the SALSA feature representation is obtained [6].

**Inter-channel phase difference (IPD)** is a classical spatial feature that has been investigated in sound source localization, separation, and enhancement tasks [25]. The IPD at each time-frequency bin $(f, t)$ is computed as the angle difference between complex STFTs of microphone pairs $\mathbf{x}_{2:M,ft}$ with respect to a reference channel $x_{1,ft}^*$ where $\mathbf{x}_{ft} = [x_{1,ft}, \mathbf{x}_{2:M,ft}]^\top$:

$$ \mathbf{\Lambda}_{ft} = \arg[x_{1,ft}^* \mathbf{x}_{2:M,ft}]. \quad (3) $$

Due to its ability to encode inter-microphone time delays in the frequency domain, IPD provides reliable spatial cues even under reverberant or noisy conditions, stacked with magnitude-based spectral features in sound localization tasks [26].

**Normalized inter-channel phase difference (NIPD)** is the frequency normalized IPD given by:

$$ \mathbf{\Phi}_{ft} = -c(2\pi f)^{-1} \mathbf{\Lambda}_{ft}. \quad (4) $$

Similar to SALSA, SALSA-Lite stacks the NIPD with linear-frequency log-power spectrograms. Compared to the EPV used in SALSA, the NIPD significantly reduces computational cost by avoiding the singular value decomposition (SVD) required to obtain eigenvectors. As shown in the study of SALSA-Lite [7], its superior performance compared to other models is attributed to the time-frequency consistency between spectral and phase difference features, which helps the model better capture sound events without being confused by inconsistent time and frequency bins.

### 2.2. Output features

Similar to classic classification tasks, the SED head usually outputs binary or multi-label indicators for the presence of sound events over time. Each output dimension corresponds to a predefined sound class and is typically activated by a sigmoid function, with binary cross-entropy loss used for training.

**Location classification** discretizes the 2D or 3D space (e.g., azimuth/elevation grid) and treats the DOA estimation as a classification task [20], [27]–[29].

**Location regression** directly regresses the continuous DOA coordinates (e.g., polar (azimuth, elevation) or Cartesian $(x, y, z)$) via fully connected layers and is typically trained using mean squared error (MSE) or angular loss [6], [7], [23]. This enables fine-grained localization but may be less robust in multi-source scenarios if event activity is not properly decoupled.

**ACCDOA** format, as applied in [23], [26], [30], unifies SED and DOA estimation in a single 3D output vector per class per frame. For each sound class and time frame, the network predicts a vector $\mathbf{v} = (x, y, z)$. If the event is inactive, $\mathbf{v} = (0, 0, 0)$; if it is active, $\mathbf{v}$ encodes the DOA direction on the unit sphere.

### 2.3. Acoustic priors in SELD

Although previous studies have investigated using RIR datasets for data augmentation [22], [23], and using SCM priors in source separation and jointly with localization through probabilistic modeling [11], [31], [32], to the best of the authors' knowledge, the direct integration of acoustic priors into SELD network objectives remains underexplored. In this work, we propose to regularize SELD training using a theoretical SCM derived from known or estimated RIR, aligning model predictions with the physical room and array geometry.

## 3. PROPOSED METHOD

In this paper, we employ a DNN $f: \mathbf{\Phi}(\mathbf{X}) \mapsto (\hat{\mathbf{Y}}_{\text{SED}}, \hat{\mathbf{Y}}_{\text{DOA}})$ that takes the SALSA-Lite, which is a stack of the scaled spectrogram and the NIPD given by Eq. (4), as the input $\mathbf{\Phi}(\mathbf{X})$ and outputs $\hat{\mathbf{Y}}_{\text{SED}} \in [0, 1]^{C \times T}$ representing the frame-wise event probabilities for $C$ classes and $\hat{\mathbf{Y}}_{\text{DOA}} \in \mathbb{R}^{C \times 3 \times T}$ indicating a 3-D direction vector for each class $c$ at frame $t$, as in the ACCDOA format. This section introduces the physically informed spatial regularization used for training the DNN.

### 3.1. Probabilistic model

We assume that the empirical (observed) SCM $\tilde{\mathbf{G}}_{ft} \in \mathbb{C}^{M \times M}$ in Eq. (2) and the $n$-th source SCM $\tilde{\mathbf{G}}_{nft} \in \mathbb{C}^{M \times M}$ follow a complex-valued inverse Wishart distribution, i.e.,

$$ \tilde{\mathbf{G}}_{ft} \sim \mathcal{I}W_{\mathbb{C}}(\tilde{\mathbf{G}}_{ft} | \nu, \mathbf{G}_{ft}), \quad (5) $$

$$ \tilde{\mathbf{G}}_{nft} \sim \mathcal{I}W_{\mathbb{C}}(\tilde{\mathbf{G}}_{nft} | \nu_n, \mathbf{G}_{nft}), \quad (6) $$

where $\mathcal{I}W_{\mathbb{C}}(\tilde{\mathbf{G}}|\nu, \mathbf{G}) \propto \det(\tilde{\mathbf{G}})^{-(\nu+M)} \det(\mathbf{G})^{\nu} \text{etr}(-\mathbf{G}\tilde{\mathbf{G}}^{-1})$ with $\text{etr}(\cdot)$ is the exponential of a trace and the degree of freedom $\nu$ determines the uncertainty of the prior distribution and could be determined based on the number of samples (frames) used for computing an SCM. This inverse Wishart distribution is widely used as a conjugate prior for covariance matrices in multivariate complex Gaussian models for source separation and localization [11], [33]. In this paper, we use the inverse Wishart distribution whose scale parameter is obtained given an estimated DOA (as illustrated in Fig. 1) to represent a physically informed prior information to regularize the DNN training.

Given the DNN output $\hat{\mathbf{Y}}_{\text{DOA}}$ at time frame $t$, we can construct a set of algebraically computed or retrieved steering vectors (SVs) denoted by $K_{ft} \triangleq \{\mathbf{h}_{nft}\}_{n=1}^{N_t}$, where each $\mathbf{h}_{nft} \in \mathbb{C}^M$ corresponds
to an estimated sound source direction. The number of SVs in $K_{ft}$ denoted by $N_t$ varies according to the number of activated sources at frame $t$. The scale parameter for the $n$-th source can then be computed as $\mathbf{G}_{nft} = \mathbf{h}_{nft}\mathbf{h}_{nft}^{\mathrm{H}} \in \mathbb{C}^{M \times M}$.

We assume that the empirical SCM is a sum of the source SCMs as $\tilde{\mathbf{G}}_{ft} = \sum_{n=1}^{N_t} \tilde{\mathbf{G}}_{nft}$. The first-moment matching technique gives us the relation between the distribution parameters:

$$ \mathbb{E}[\tilde{\mathbf{G}}_{ft}] = \sum_{n=1}^{N_t} \mathbb{E}[\tilde{\mathbf{G}}_{nft}] \Leftrightarrow \frac{\mathbf{G}_{ft}}{\nu-M} = \sum_{n=1}^{N_t} \frac{\mathbf{G}_{nft}}{\nu_n-M} \quad (7) $$

We then assume that $\nu_n$ and $\nu$ are all equal, so $\mathbf{G}_{ft} = \sum_{n=1}^{N_t} \mathbf{G}_{nft}$. Therefore, given the DNN output $\hat{\mathbf{Y}}_{\mathrm{DOA}}$ at time frame $t$, we can also obtain the scale parameter for the observation $\mathbf{G}_{ft}$.

## 3.2. Source spatial covariance matrix parameter estimation

We propose below two methods of estimating the scale parameter $\mathbf{G}_{nft}$ given the set of SVs $K_{ft}$. The SVs are computed algebraically in the first method, while in the second method, we retrieve ones from a set of measured RIRs. Both methods are compatible with the location regression and ACCDOA output formats.

**Algebraic steering-based SCM.** To model the ideal spatial and phase relationships across microphones given a known microphone geometry, $\mathbf{G}_{nft}$ can be obtained from the algebraic SVs $\mathbf{h}_{nft} \in \mathbb{C}^M$ as

$$ \mathbf{G}_{nft} = \mathbf{h}_{nft}\mathbf{h}_{nft}^{\mathrm{H}} + \epsilon\mathbf{I}, \quad (8) $$

where $\epsilon > 0$ is added to ensure that $\mathbf{G}_{nft}$ is positive definite and $\mathbf{h}_{nmft} \triangleq \exp(-2j\pi\omega_f\mathbf{u}_{nt}^{\mathrm{T}}\mathbf{q}_m/c)$ is the component of $\mathbf{h}_{nft}$ corresponding to the $m$-th microphone with $f$ is the frequency in Hz, $c \in \mathbb{R}$ denotes the speed of sound, $j$ is the imaginary unit, $\mathbf{u}_{nt}^{\mathrm{T}}$ denotes the transposition, $\mathbf{u}_{nt} \in \mathbb{R}^3$ is a unit-length vector indicating the for source index $n$ in 3D Cartesian coordinates at frame $t$, and $\mathbf{q}_m \in \mathbb{R}^3$ denotes the position vector of the $m$-th microphone relative to a reference point, i.e., the array center.

**RIR retrieved steering-based SCM.** Since the algebraic steering vector-based SCM only encodes the direct path and cannot integrate a detailed reflection and reverberation information, we further assume access to a set of measured RIRs $\mathcal{R} \triangleq \{\mathbf{R}_q \in \mathbb{R}^{L \times M}\}_{q=1}^Q$, where $Q, L$ denote the number of available RIRs and their length, respectively. For the sake of clarity, we hereafter fix a time frame $t$ and design $\hat{\mathbf{Y}}_{\mathrm{DOA}} \triangleq \{\hat{\mathbf{Y}}_{\mathrm{DOA},n} \in \mathbb{R}^3\}_{n=1}^{N_t}$ the DOA output that contains $N_t \le C$ classes. For each source index $n$, we retrieve $M$-channel RIR candidate whose direct-path direction is the closest (highest cosine similarity) to $\hat{\mathbf{Y}}_{\mathrm{DOA},n}: \mathbf{H}_n = \mathcal{D}(\hat{\mathbf{Y}}_{\mathrm{DOA},n}, \mathcal{R}) \in \mathbb{R}^{L \times M}$ where $\mathcal{D}$ denotes this nearest-neighbor operator. Next, for each column $\mathbf{h}_{nm} \in \mathbb{R}^L$ of $\mathbf{H}_n$ that represents the candidate RIR at position $n$ captured by microphone $m$, we compute $\mathbf{h}_{nm} \triangleq \{\hat{\mathbf{h}}_{nmf\tau}\}_{f,\tau=1}^{F,\tau=1}$ where $\hat{\mathbf{h}}_{nmf\tau} \triangleq \mathrm{STFT}(\mathbf{w} \star \mathbf{h}_{nm})$ represents the $(f, \tau)$-STFT coefficient of $\mathbf{h}_{nm}$ convolved with a Gaussian white noise $\mathbf{w}$ of length $L$. Stacking microphone channels gives $\hat{\mathbf{h}}_{n\tau} \triangleq [\hat{h}_{n1\tau}, \dots, \hat{h}_{nM\tau}]^\mathrm{T} \in \mathbb{C}^M$. The per-frame SCM is given by

$$ \mathbf{G}_{nft} = \mathbb{E}_{\tau} \left[ \hat{\mathbf{h}}_{n\tau} \hat{\mathbf{h}}_{n\tau}^{\mathrm{H}} \right]. \quad (9) $$

## 3.3. Proposed loss function

We aim to maximize the inverse Wishart log-likelihood during training, so the final loss function to be minimized is given by

$$ \mathcal{L} = \lambda \mathcal{L}_{\mathrm{BCE}}(\hat{\mathbf{Y}}_{\mathrm{SED}}, \mathbf{Y}_{\mathrm{SED}}) \quad (10) $$

$$ + \gamma \mathbf{1}_{\mathrm{active}} \mathcal{L}_{\mathrm{MSE}}(\hat{\mathbf{Y}}_{\mathrm{DOA}}, \mathbf{Y}_{\mathrm{DOA}}) \quad (11) $$

$$ - \alpha \mathbb{E}_{f,t \in \mathcal{I}_{\mathrm{active}}} \log \mathcal{I}W_C(\tilde{\mathbf{G}}_{ft} | \nu, \mathbf{G}_{ft}), \quad (12) $$

where $\lambda, \gamma, \alpha$ are the weights of different loss terms, the binary cross-entropy loss $\mathcal{L}_{\mathrm{BCE}}$ is used to evaluate the event detection given the estimated and ground truth event classes denoted by $\hat{\mathbf{Y}}_{\mathrm{SED}}$ and $\mathbf{Y}_{\mathrm{SED}}$, respectively, and the mean squared error $\mathcal{L}_{\mathrm{MSE}}$ is used to measure the accuracy of the DOA estimation given the estimated and ground truth DOAs denoted by $\hat{\mathbf{Y}}_{\mathrm{DOA}}$ and $\mathbf{Y}_{\mathrm{DOA}}$, respectively. The scale parameter $\mathbf{G}_{ft}$ is computed given the DNN output $\hat{\mathbf{Y}}_{\mathrm{DOA}}$ as described in Sec. 3.1 and to deal with non-stationary sound sources, the empirical SCM $\tilde{\mathbf{G}}_{ft}$ is computed using Eq. (2) by averaging across a few neighboring frames. Additionally, we consider a set of time frames with active events $\mathcal{I}_{\mathrm{active}} \subset \{1, \dots, T\}$ and use it to avoid the detrimental impact of SCMs with inactive events. Similarly, a binary mask $\mathbf{1}_{\mathrm{active}} \in \{0, 1\}^T$ is used to selectively apply the mean squared error only to active frames.

# 4. EXPERIMENTS & RESULTS

## 4.1. Experimental setup

**Dataset.** A synthetic dataset is built in this study with the sound event dataset NIGENS [34] and the RIR database TAU Spatial Room Impulse Response (TAU-SRIR) [35] to carry out the RIR prior to informed training. The mixtures are created by convolving isolated sound sources from the NIGENS dataset with RIR measurements in nine rooms provided by TAU-SRIR. Spatial noise from TAU-SRIR is also mixed in synthetic recording, and the overall generation process follows the same procedure as that for the TAU-NIGENS dataset [36], ensuring consistency and comparability with prior work. For each room, 50 mixture recordings are generated, and each mixture has a fixed duration of 60 seconds under a sample rate of 24 kHz and features moving sources. Rooms 1 to 6 and 8 are used for training and 10% of them are used in in-domain evaluation. Mixtures and RIRs from rooms 9 and 10 are isolated from the training set and these mixtures from unseen rooms serve as out-of-domain data for validation and final out-of-domain evaluation, aiming at testing the domain adaptation ability of the model in new acoustic conditions.

**Training and model setting.** We adopt the SALSA-Lite architecture [7] and train the baseline for SED and DOA prediction using its original loss. The proposed method builds on this baseline by incorporating the additional regularization term defined in Eq. (12), where the degree of freedom is set to $\nu = 6$ and the number of channels is $M = 4$. The weights for the loss function and regularization term are $\lambda = 0.3, \gamma = 0.7$, and $\alpha = 0.5 \times 10^{-6}$. The algebraic SVs are obtained with known microphone array geometry in TAU-SIR [35], and the RIR-based steering vector is retrieved and computed from the room corresponding to each mixture. The baseline model trained with inverse Wishart regularization based on each is denoted by **IW-Algebraic** and **IW-RIR** in experimental results. The empirical SCM is computed with over 9 neighboring frames to reduce temporal fluctuation, and the SCMs are obtained by the mixture covariance normalized by trace. **Evaluation metrics.** Four metrics are reported in this study: location-dependent error rate ($\mathrm{ER}_{\le 20^{\circ}}$), F1 score ($\mathrm{F}_{\le 20^{\circ}}$) for SED, class-dependent localization error ($\mathrm{LE}_{\mathrm{CD}}$) and localization recall ($\mathrm{LR}_{\mathrm{CD}}$) for DOA estimation. Following the baseline model evaluation [7], an aggregated score $\mathcal{E}_{\mathrm{SELD}}$ is computed as

$$ \mathcal{E}_{\mathrm{SELD}} = \frac{1}{4} \left[ \mathrm{ER}_{\le 20^{\circ}} + (1 - \mathrm{F}_{\le 20^{\circ}}) + \frac{\mathrm{LE}_{\mathrm{CD}}}{180^{\circ}} + (1 - \mathrm{LR}_{\mathrm{CD}}) \right]. \quad (13) $$

## 4.2. Cross-acoustic environment performance

The results of the baseline model in-domain and out-of-domain settings show that unseen acoustic environments have an impact on model performance in Table 1. Relying on learned reverberation and reflections
<figure><img src="image_5.png" /><figcaption>Fig. 2: Model performance across acoustic environments (R1: Large classroom, R2: Gym, R3: Bomb shelter, R4: Meeting room, R5: Lecture hall). Reverberation time RT₆₀ (s): R1 = 0.98, R2 = 0.74, R3 = 1.22, R4 = 0.60, R5 = 0.86. Estimated using the Schroeder integration method [37].</figcaption></figure>

<table><caption>Table 1: Experimental results on in-domain and out-of-domain evaluation. In-domain evaluation refers to testing on isolated mixtures from the rooms learned during training, and out-of-domain evaluation uses recordings from unseen acoustic environments. Low-data refers to training with only 10% of the full dataset, alongside the corresponding percentage degradation compared to full-data training. Lower degradation values indicate greater robustness under data scarcity.</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="5">In-domain Evaluation</th><th colspan="5">Out-of-domain Evaluation</th></tr><tr><th>ε<sub>SELD</sub><sup>↓</sup></th><th>ER<sub>≤20°</sub><sup>↓</sup></th><th>LE<sub>CD</sub><sup>↓</sup></th><th>F<sub>≤20°</sub><sup>↑</sup></th><th>LR<sub>CD</sub><sup>↑</sup></th><th>ε<sub>SELD</sub><sup>↓</sup></th><th>ER<sub>≤20°</sub><sup>↓</sup></th><th>LE<sub>CD</sub><sup>↓</sup></th><th>F<sub>≤20°</sub><sup>↑</sup></th><th>LR<sub>CD</sub><sup>↑</sup></th></tr></thead><tbody><tr><td rowspan="3">Baseline [7]</td><td>Full-data</td><td>0.323</td><td>0.498</td><td>35.195</td><td>0.636</td><td>0.765</td><td><strong>0.485</strong></td><td>0.703</td><td><strong>31.389</strong></td><td><strong>0.428</strong></td><td>0.510</td></tr><tr><td>IW-Algebraic</td><td>0.323</td><td>0.489</td><td>34.614</td><td>0.636</td><td>0.752</td><td>0.488</td><td><strong>0.702</strong></td><td>35.410</td><td>0.414</td><td><strong>0.533</strong></td></tr><tr><td>IW-RIR</td><td><strong>0.314</strong></td><td><strong>0.484</strong></td><td><strong>33.968</strong></td><td><strong>0.646</strong></td><td><strong>0.771</strong></td><td>0.494</td><td>0.714</td><td>34.172</td><td>0.411</td><td>0.516</td></tr><tr><td rowspan="3">Baseline [7]</td><td>Low-data</td><td>0.537</td><td>0.747</td><td>46.537</td><td>0.324</td><td>0.534</td><td>0.652</td><td>0.851</td><td>47.964</td><td>0.180</td><td>0.329</td></tr><tr><td>IW-Algebraic</td><td>0.536</td><td>0.747</td><td>46.171</td><td>0.326</td><td>0.535</td><td><strong>0.631</strong></td><td>0.836</td><td>41.212</td><td><strong>0.208</strong></td><td><strong>0.331</strong></td></tr><tr><td>IW-RIR</td><td><strong>0.531</strong></td><td><strong>0.742</strong></td><td><strong>44.654</strong></td><td><strong>0.331</strong></td><td><strong>0.534</strong></td><td>0.632</td><td><strong>0.832</strong></td><td><strong>40.020</strong></td><td>0.203</td><td>0.322</td></tr><tr><td rowspan="3">Baseline [7]</td><td>Degradation</td><td>66.2%</td><td><strong>50.0%</strong></td><td>32.2%</td><td>49.0%</td><td>30.1%</td><td>34.4%</td><td>21.0%</td><td>52.8%</td><td>57.9%</td><td><strong>35.4%</strong></td></tr><tr><td>IW-Algebraic</td><td>Degradation</td><td><strong>65.9%</strong></td><td>52.8%</td><td>33.4%</td><td><strong>48.7%</strong></td><td><strong>28.9%</strong></td><td>29.3%</td><td>19.1%</td><td><strong>16.4%</strong></td><td><strong>49.8%</strong></td><td>37.9%</td></tr><tr><td>IW-RIR</td><td>Degradation</td><td>69.1%</td><td>53.3%</td><td><strong>31.5%</strong></td><td>48.8%</td><td>30.7%</td><td><strong>27.9%</strong></td><td><strong>16.5%</strong></td><td>17.1%</td><td>50.6%</td><td>37.6%</td></tr></tbody></table>

from in-domain training recordings, it notably affects the localization performance. With the addition of proposed regularization, especially the RIR prior, the localization error of the model is improved notably in the learned acoustic environment. Fig. 3 illustrates the F1 score versus the error rate under different thresholds, and the proposed regularization consistently performs slightly better across all thresholds. Note that one close to the top-left indicates better performance, as it corresponds to higher F1 scores and lower error rates under varying DOA criteria. We further investigate the model’s performances across different acoustic environments, as shown in Fig. 2. In rooms with moderate reverberation (R2, R4, R5), both proposed methods generally outperform the baseline, especially in terms of the localization error. But the RIR-based method underperforms in highly reverberant rooms (R1, R3), because the limited RIR length may fail to capture very-late decay components. The algebraic steering vector method shows better robustness to reverberation than the RIR-based approach, achieving a clear reduction in LE<sub>CD</sub> in the most reverberant room (R3), though its performance on the other metrics remains limited. This finding informs that this regularization present limitations lie in applying to severely low direct-to-reverberant ratio or highly reverberant environments.

### 4.3. Low-data regime performance

As shown in other work [21], the introduction of physically based priors in deep learning has the potential to increase the robustness of the method in a low-data regime. To evaluate the robustness and data efficiency of the proposed method, we conduct experiments using only 10% of the training data from the aforementioned dataset. The model is trained solely on this reduced subset, while other parameters and the evaluation procedure remain unchanged and follow the same procedure as in the full-data setting. Table 1 also shows the proposed

<figure><img src="image_6.png" /><figcaption>Fig. 3: F1-score versus error rate across varying DOA thresholds on full-data (Left) and low-data regime (Right).</figcaption></figure>

method, which is still valid and robust in this training regime, with a smaller percentage degradation than the baseline. Notably, our method in a low-data setting yields better out-of-domain generalization than the full-data counterpart, suggesting that future work may leverage physical priors instead of indiscriminately enlarging the training set.

## 5. CONCLUSION

We proposed a physically informed regularization method for deep learning-based SELD models that incorporates probabilistic modeling of microphone array geometry and RIR priors. Such a technique is applied only during training and does not introduce additional complexity at inference time. Our method improves performance over baseline models without regularization, across both full-scale and limited-data settings. Future work will explore the use of measured steering vectors with interpolation based on neural field theory [38] and an extension to self-supervised learning, since the proposed loss function can be computed without ground truth.
REFERENCES

[1] A. Politis, A. Mesaros, S. Adavanne, T. Heittola, and T. Virtanen, "Overview and evaluation of sound event localization and detection in DCASE 2019," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 29, pp. 684–698, 2020.

[2] T. Hayashi, T. Komatsu, R. Kondo, T. Toda, and K. Takeda, "Anomalous sound event detection based on WaveNet," in *EUSIPCO*. IEEE, 2018, pp. 2494–2498.

[3] M. Crocco, M. Cristani, A. Trucco, and V. Murino, "Audio surveillance: A systematic review," *ACM Computing Surveys (CSUR)*, vol. 48, no. 4, pp. 1–46, 2016.

[4] J. W. Yeow, E.-L. Tan, J. Bai, S. Peksi, and W.-S. Gan, "Real-time sound event localization and detection: Deployment challenges on edge devices," *arXiv preprint arXiv:2409.11700*, 2024.

[5] Y. He, N. Trigoni, and A. Markham, "SoundDet: Polyphonic moving sound event detection and localization from raw waveform," in *Proc. ICML*. PMLR, 2021, pp. 4160–4170.

[6] T. N. T. Nguyen, K. N. Watcharasupat, N. K. Nguyen, D. L. Jones, and W.-S. Gan, "SALSA: Spatial cue-augmented log-spectrogram features for polyphonic sound event localization and detection," *IEEE/ACM Trans. Audio, Speech, Lang. Process.*, vol. 30, pp. 1749–1762, 2022.

[7] T. N. T. Nguyen, D. L. Jones, K. N. Watcharasupat, H. Phan, and W.-S. Gan, "SALSA-Lite: A fast and effective feature for polyphonic sound event localization and detection with microphone arrays," in *Proc. ICASSP*. IEEE, 2022, pp. 716–720.

[8] T. Khandelwal, R. K. Das, E. S. Chng et al., "Sound event detection: a journey through DCASE challenge series," *APSIPA Transactions on Signal and Information Processing*, vol. 13, no. 1, 2024.

[9] G. Chardon, "Gridless covariance matrix fitting methods for three dimensional acoustical source localization," *Journal of Sound and Vibration*, vol. 551, p. 117608, 2023.

[10] J. Nikunen and T. Virtanen, "Direction of arrival based spatial covariance model for blind sound source separation," *IEEE/ACM Trans. Audio, Speech, Lang. Process.*, vol. 22, no. 3, pp. 727–739, 2014.

[11] Y. Sumura, D. D. Carlo, A. A. Nugraha, Y. Bando, and K. Yoshii, "Joint audio source localization and separation with distributed microphone arrays based on spatially-regularized multichannel NMF," in *2024 18th International Workshop on Acoustic Signal Enhancement (IWAENC)*, 2024, pp. 145–149.

[12] S. Gannot, E. Vincent, S. Markovich-Golan, and A. Ozerov, "A consolidated perspective on multimicrophone speech enhancement and source separation," *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, vol. 25, no. 4, pp. 692–730, 2017.

[13] A. Politis, S. Tervo, and V. Pulkki, "COMPASS: Coding and multidirectional parameterization of ambisonic sound scenes," in *Proc. ICASSP*, 2018, pp. 6802–6806.

[14] N. Q. Duong, E. Vincent, and R. Gribonval, "Spatial location priors for gaussian model based reverberant audio source separation," *EURASIP J. Adv. Signal Process.*, vol. 2013, pp. 1–11, 2013.

[15] R. Falcon-Perez, G. Wichern, F. G. Germain, and J. Le Roux, "Location as supervision for weakly supervised multi-channel source separation of machine sounds," in *Proc. WASPAA*. IEEE, 2023, pp. 1–5.

[16] K. Kamo, Y. Kubo, N. Takamune, D. Kitamura, H. Saruwatari, Y. Takahashi, and K. Kondo, "Joint-diagonalizability-constrained multichannel nonnegative matrix factorization based on multivariate complex subgaussian distribution," in *EUSIPCO*. IEEE, 2021, pp. 890–894.

[17] N. Q. Duong, E. Vincent, and R. Gribonval, "Under-determined reverberant audio source separation using a full-rank spatial covariance model," *IEEE Transactions on Audio, Speech, and Language Processing*, vol. 18, no. 7, pp. 1830–1840, 2010.

[18] Y. Shao, S.-X. Zhang, and D. Yu, "RIR-SF: Room impulse response based spatial feature for target speech recognition in multi-channel multi-speaker scenarios," in *Proc. Interspeech*, 2024, pp. 4988–4992.

[19] D. Berghi and P. J. Jackson, "Reverberation-based features for sound event localization and detection with distance estimation," *arXiv preprint arXiv:2504.08644*, 2025.

[20] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen, "Sound event localization and detection of overlapping sources using convolutional recurrent neural networks," *IEEE Journal of Selected Topics in Signal Processing*, vol. 13, no. 1, pp. 34–48, 2018.

[21] N. Sato, M. Yasuda, S. Saito, and N. Harada, "Sound source distance estimation utilizing physics-informed prior for sound event localization and detection," in *Proc. ICASSP*. IEEE, 2025, pp. 1–5.

[22] C. Ick and B. McFee, "Leveraging geometrical acoustic simulations of spatial room impulse responses for improved sound event detection and localization," *arXiv preprint arXiv:2309.03337*, 2023.

[23] J. S. Kim, H. J. Park, W. Shin, and S. W. Han, "A robust framework for sound event localization and detection on real recordings," *Tech. Rep.*, 2022.

[24] S. Koyama, E. De Sena, P. Samarasinghe, M. R. Thomas, and F. Antonacci, "Past, present, and future of spatial audio and room acoustics," in *Proc. ICASSP*. IEEE, 2025, pp. 1–5.

[25] T. Yoshioka and T. Nakatani, "Generalization of multi-channel linear prediction methods for blind mimo impulse response shortening," *IEEE Trans. Audio, Speech, Lang. Process.*, vol. 20, no. 10, pp. 2707–2720, 2012.

[26] K. Shimada, Y. Koyama, S. Takahashi, N. Takahashi, E. Tsunoo, and Y. Mitsufuji, "Multi-ACCDOA: Localizing and detecting overlapping sounds from the same class with auxiliary duplicating permutation invariant training," in *Proc. ICASSP*, 2022, pp. 316–320.

[27] L. Wang, Z. Jiao, Q. Zhao, J. Zhu, and Y. Fu, "Framewise multiple sound source localization and counting using binaural spatial audio signals," in *Proc. ICASSP*. IEEE, 2023, pp. 1–5.

[28] U. Kowalk, S. Doclo, and J. Bitzer, "Geometry-aware DOA estimation using a deep neural network with mixed-data input features," in *Proc. ICASSP*. IEEE, 2023, pp. 1–5.

[29] P.-A. Grumiaux, S. Kitić, P. Srivastava, L. Girin, and A. Guérin, "SAL-ADnet: Self-attentive multisource localization in the Ambisonics domain," in *Proc. WASPAA*. IEEE, 2021, pp. 336–340.

[30] Y. Cao, T. Iqbal, Q. Kong, F. An, W. Wang, and M. D. Plumbley, "An improved event-independent network for polyphonic sound event localization and detection," in *Proc. ICASSP*. IEEE, 2021, pp. 885–889.

[31] H. Munakata, Y. Bando, R. Takeda, K. Komatani, and M. Onishi, "Joint separation and localization of moving sound sources based on neural full-rank spatial covariance analysis," *IEEE Signal Processing Letters*, vol. 30, pp. 384–388, 2023.

[32] Y. Nozaki, Y. Bando, and M. Onishi, "Source-aware spatial self-supervision for sound event localization and detection," in *Proc. ICASSP*, 2025, pp. 1–5.

[33] J. Azcarreta, N. Ito, S. Araki, and T. Nakatani, "Permutation-free cGMM: Complex Gaussian mixture model with inverse Wishart mixture model based spatial prior for permutation-free source separation and source counting," in *Proc. ICASSP*, 2018, pp. 51–55.

[34] I. Trowitzsch, J. Taghia, Y. Kashef, and K. Obermayer, "The NIGENS general sound events database," *arXiv preprint arXiv:1902.08314*, 2019.

[35] A. Politis, S. Adavanne, and T. Virtanen, "TAU spatial room impulse response database (TAU-SRIR dB)," 2022.

[36] —, "A dataset of reverberant spatial sound scenes with moving sources for sound event localization and detection," *arXiv preprint arXiv:2006.01919*, 2020.

[37] M. R. Schroeder, "New method of measuring reverberation time," *The Journal of the Acoustical Society of America*, vol. 37, no. 3, pp. 409–412, 1965.

[38] D. Di Carlo, A. A. Nugraha, M. Fontaine, Y. Bando, and K. Yoshii, "Neural steerer: Novel steering vector synthesis with a causal neural field over frequency and direction," in *Proc. ICASSP*, 2024.