SPEAR: Receiver-to-Receiver Acoustic Neural Warping Field

## Abstract

Abstract We present SPEAR , a continuous receiver-to-receiver acoustic neural warping field for spatial acoustic effects prediction in an acoustic 3D space with a single stationary audio source. Unlike traditional source-to-receiver modelling methods that require prior space acoustic properties knowledge to rigorously model audio propagation from source to receiver, we propose to predict by warping the spatial acoustic effects from one reference receiver position to another target receiver position, so that the warped audio essentially accommodates all spatial acoustic effects belonging to the target position. SPEAR can be trained in a data much more readily accessible manner, in which we simply ask two robots to independently record spatial audio at different positions. We further theoretically prove the universal existence of the warping field if and only if one audio source presents. Three physical principles are incorporated to guide SPEAR network design, leading to the learned warping field physically meaningful. We demonstrate SPEAR superiority on both synthetic, photo-realistic and real-world dataset, showing the huge potential of SPEAR to various down-stream robotic tasks.

## 1 Introduction

In an enclosed acoustic 3D space where a stationary sound source keeps emitting spatial audio, the primary objective is to precisely delineate the spatial acoustic effects for any given receiver position. These spatial acoustic effects typically encompass reverberation, loudness variation and resonance. Achieving high-fidelity and authentic spatial acoustic effect modelling is pivotal for delivering a truly immersive 3D acoustic experience that seamlessly integrates with the 3D room scene. Consequently, such modelling techniques have a wide range of applications in auditory AR/VR techniques [, , ], audio-inclusive robot tasks [] and reconstruction endeavors [, ].

To model the spatial acoustic effects, most prior methods [, , , , , , , , ] follow the source-to-receiver pipeline to explicitly model sound propagation process from source to receiver, where the overall behavioural change along the propagation path is usually described by room impulse response (RIR). As the spatial audio propagates in a complex way encompassing diffraction, reflection and absorption, the resulting RIR is highly non-smooth and lengthy in data points. Classic methods, either wave-based [, , , ] and geometry-based modelling [], require massive prior knowledge of the 3D space’s acoustic properties such as source position, space geometric layout and constructional material to precisely simulate the propagation process for a given source-receiver pair. However, accessing such prior knowledge poses a formidable challenge in reality and the whole computation is inextricably inefficient. Some recent work (e.g., NAF []) aim to alleviate this computational burden by learning a continuous acoustic neural field. Nonetheless, training such a continuous field necessitates vast RIR data which is exceedingly difficult to collect in real scenarios.

In this work, we instead propose to predict spatial acoustic effects from receiver-to-receiver perspective. Our framework, termed *SPEAR*, relies on neither RIR data nor prior space acoustic properties that are difficult to obtain and required by existing source-to-receiver based methods [, , ], but instead simply require much more readily accessible data – the receiver recorded audio at discrete positions. Our observation is that directly carrying a receiver (can ask robot or human to hold the receiver) to record audio at different positions is much readily executable than measuring RIR data and space acoustic properties. Since receiver recorded audio naturally encodes the spatial acoustic effects at its position, analyzing the received audio can help to acoustically characterize the 3D space, and further predict the spatial acoustic effects for any given novel receiver position.

Figure: Figure 1: SPEAR Motivation: A stationary audio source is emitting audio in 3D space. Requiring neither source position nor 3D space acoustic properties, *SPEAR* simply requires two microphones to actively record the spatial audio independently at discrete positions. During training, *SPEAR* takes as input a pair of receiver positions and outputs a warping field potentially warping the recorded audio on reference position to target position. Minimizing the discrepancy between the warped audio and recorded audio enforces *SPEAR* to acoustically characterise the 3D space from receiver-to-receiver perspective. The learned *SPEAR* is capable of predicting spatial acoustic effects at arbitrary positions.
Refer to caption: x1.png

As is shown in Fig. 1, to obtain the training data, we simply require receivers to record the audio in the 3D space independently. At each discrete position pair, the two receivers are temporally synchronized to record the same audio content and their respective positions are recorded as well. *SPEAR* then learns a continuous receiver-to-receiver acoustic neural warping field that takes as input two receivers’ position and outputs a neural warping field warping the spatial acoustic effects from one reference position to the other target position. With the learned *SPEAR*, we can warp the audio recorded at one arbitrary reference position to another arbitrary target position so that the warped audio fully accommodates the spatial acoustic effects at the target position. *SPEAR* has huge potential in various robotic tasks, such as audio-involved robot relocalization and manipulation.

We further theoretically prove the universal existence of the receiver-to-receiver warping field if and only if one stationary audio source presents in the 3D space, then we introduce three main physical principles underpinned by linear time-invariant (LTI) 3D acoustic space that guide *SPEAR* neural network design: *Globality*, *Order Awareness* and *Audio-Content Agnostic*. We adopt the Transformer [] architecture to predict the warping field in frequency domain, where the lengthy warping field is divided into small and non-overlapping patches. Each token is responsible for predicting a patch. We run experiments on both synthetic, photo-realistic and real-world datasets to show the superity of *SPEAR*. In summary, we make three main contributions:

- 1.
We propose *SPEAR*, a novel receiver-to-receiver spatial acoustic effects prediction framework. Unlike existing source-to-receiver modelling methods requiring extensive prior space acoustic properties knowledge, *SPEAR* can be efficiently trained in a data more readily accessible manner.
- 2.
We theoretically prove the universal existence of receiver-to-receiver neural warping field if and only if one stationary audio source presents in the 3D space. *SPEAR* network design is based on three acoustic physical principles, so that the whole neural network is physically meaningful.
- 3.
We demonstrate *SPEAR* superiority on both synthetic data, photo-realistic and real-world dataset.

## 2 Related Work

Spatial Acoustic Effects Modelling. Classic methods tend to numerically compute room impulse response (RIR). They can be divided into two main categories: wave-based [, , , ] and geometry-based (aka geometrical acoustics) [, , , , , ]. They require extensive prior knowledge of the 3D space acoustic properties such as the audio source position [, ], room geometry and furniture arrangement to derive RIR. Moreover, they are computationally expensive and the whole computation needs to be resumed once either the source or receiver position gets changed. Our proposed *SPEAR* circumvents the dependency on extensive prior knowledge and learns the spatial acoustic effects in a data more readily accessible manner. The advent of deep neural networks has inspired recent work [, , , , , , , , , ] to focus on learning RIR with deep neural networks. While showing promising performance, they still require massive RIR data or even crossdomain visual data [] to train their model, which in reality are difficult to collect. While all of those methods fall into source-to-receiver estimation, *SPEAR* infers spatial acoustic effects from a receiver-to-receiver perspective which naturally brings several advantages over existing methods.

Neural Implicit Representation. Implicit representation learning has received lots of attention in recent years, especially in computer vision community [, , , , ]. They model static or dynamic visual scenes by optimizing an implicit neural radiance field in order to render photo-realistic novel views. Some recent work (e.g., NAF [], FewShotRIR []) propose to learn implicit neural acoustic fields from source-receiver pairs or audio-visual cues. *SPEAR* also learns a spatial continuous neural implicit representation for spatial acoustic effects prediction.

Audio Synthesis. Estimating the audio for at a novel position partially relates to audio synthesis [, , , , , , , ]. WaveNet [] learns to predict future sound waveform based on previously heard sound waveform. WaveGAN [] and GANSynth [] adopt generative adversarial network (GAN []) to learn to generate audio. Our framework *SPEAR* differs from audio synthesis as it focuses on spatial acoustic effects modelling.

Time-series Prediction. Predicting warping field in either time or frequency domain partially relates to time-series prediction. Existing deep neural network based time-series prediction methods can be divided into four main categories: Convolutional Neural Networks (CNNs) based [, , , ], Recurrent Neural Networks based [, , ], Graph Neural Networks based (GNNs) [, , , , ] and Transformer based [, , ] methods. However, unlike existing works, warping field prediction in *SPEAR* exhibits no causality and the warping field is estimated from receiver positions.

## 3 Receiver-to-Receiver Acoustic Neural Warping Field

### 3.1 Problem Formulation

In a linear time-invariant (LTI) enclosed 3D space $\mathcal{R}$, stationary sound sources are constantly emitting audio waveform. We use two receivers to record the audio at various discrete positions independently. At each time step, we temporally synchronize the two receivers before recording so that the two receivers are recording the same audio content. In addition to audio, we also record the two receivers’ spatial position. Specifically, we have collected $N$-step paired receiver dataset $\{(\mathcal{A},\mathcal{P})=\{(x_{1,i}(t),p_{1,i}),(x_{2,i}(t),p_{2,i})\}_{i=1
}^{N}\}$, $\mathcal{A}\in\mathbb{R}^{T\times N}$, $\mathcal{P}\in\mathbb{R}^{3\times N}$. The recorded audio $x_{1,i}(t)$ and $x_{2,i}(t)$ is the raw audio waveform in time domain (both are of the same length $T$ sampled at the same sampling rate). $p_{1,i}$ and $p_{2,i}$ are the two receivers’ spatial coordinate $[x_{k,i}^{\prime},y_{k,i}^{\prime},z_{k,i}^{\prime}]$ ($k\in\{1,2\}$). Our target is to learn a receiver-to-receiver acoustic neural warping field $\mathbfcal{F}$ from $(\mathcal{A},\mathcal{P})$, $\mathbfcal{F}\leftarrow(\mathcal{A},\mathcal{P})$, so that it can efficiently predict the spatial audio acoustic effects for any arbitrary target position $p_{t}$ by predicting a warping transform $\mathcal{W}_{p_{r}\rightarrow p_{t}}$ that warps audio recorded at another reference position $p_{r}$ to the target position $p_{t}$. The warped audio $\widehat{x_{p_{r}\rightarrow p_{t}}(t)}$ at position $p_{t}$ essentially accommodates the spatial acoustic effects belonging to $p_{t}$.

$$ $\mathcal{W}_{p_{r}\rightarrow p_{t}}=\mathbfcal{F}_{\theta}(p_{t},p_{r});\ \ \widehat{X_{p_{r}\rightarrow p_{t}}(f)}=\mathcal{W}_{p_{r}\rightarrow p_{t}} \cdot X_{p_{r}}(f);\ \ \ p_{t},p_{r}\notin\mathcal{P}$ (1) $$

Where $\theta$ is the trainable parameters of $\mathbfcal{F}$. $p_{r}$, $p_{t}$ are arbitrary positions in the 3D space $\mathcal{R}$. $\widehat{X_{p_{r}\rightarrow p_{t}}(f)}$ and $X_{p_{r}}(f)$ are discrete Fourier transform (DFT) representation of the warped audio at the target position and recorded audio $x_{p_{r}}(t)$ at reference position represented in time domain, respectively. For example, if $x_{p_{r}}(t)$ has $T$ points, the $T$-point DFT result $X_{p_{r}}(f)$ is a complex representation where both the real and imaginary part have $T$ data points. The learned $\mathcal{W}_{p_{r}\rightarrow p_{t}}$ is a complex representation of the same shape of $X_{p_{r}}(f)$. It is worth noting that, in our formulation, the warping transform is multiplication in frequency domain (see Sec. 3.2 for the proof), and the acoustic neural warping field is independent on audio content $\mathcal{A}$ (see Sec. 3.3 for more discussion).

To optimize $\mathbfcal{F}_{\theta}$, we minimize the discrepancy $\mathcal{L}$ between the warped audio at the target position and corresponding recorded audio,

$$ $\mathbfcal{F}_{\theta}\leftarrow\operatorname*{arg\,min}_{\theta}\ \mathcal{L} (\widehat{X_{p_{1}\rightarrow p_{2}}(f)},X_{p_{2}}(f)),\ \ \ \forall p_{1},p_{ 2}\in\mathcal{P}$ (2) $$

### 3.2 Mathematical Backend of Receiver-to-Receiver Neural Warping Field

Before introducing *SPEAR*, we need to answer two questions:

Q1:. Does the proposed warping field suffice to model receiver-to-receiver spatial acoustic effects?
Q2:. If so, is there any constraint on the audio source number and placement in the 3D space?

We first consider the simplest case where there is just one audio source in the 3D space.

Proposition 1: If the Linear Time-Invariant (LTI) 3D space contains a single audio source, then receiver-to-receiver warping exists and is uniquely defined for any pair of receiver positions.

Proof: Assume the audio source isotropically emits sound waveform $s(t)$ at a fixed position, the two receivers’ recorded spatial audio in time domain at two different positions are $x_{1}(t)$ and $x_{2}(t)$, respectively. According to room acoustics [], $x_{1}(t)$ and $x_{2}(t)$ are obtained by convolving with their respective impulse response RIR with the sound source $s(t)$,

$$ $x_{1}(t)=s(t)\circledast h_{1}(t),\ \ \ x_{2}(t)=s(t)\circledast h_{2}(t)$ (3) $$

where $h_{1}(t)$ and $h_{2}(t)$ are the two RIRs from the source $s(t)$ to receiver $x_{1}(t)$ and $x_{2}(t)$ respectively. $\circledast$ is the 1D convolution in time domain. According the Convolution theorem that time domain convolution equals to production in Frequency domain, we can rewrite Eqn. (3) as,

$$ $X_{1}(f)=S(f)\cdot H_{1}(f),\ \ \ X_{2}(f)=S(f)\cdot H_{2}(f)$ (4) $$

where $X(\cdot)$, $S(\cdot)$ and $H(\cdot)$ are the Fourier transform of receiver recorded audio, source audio and RIR, respectively. Based on Eqn. (4), we can further get,

$$ $X_{2}(f)=X_{1}(f)\cdot\frac{H_{2}(f)}{H_{1}(f)},\ \ X_{1}(f)=X_{2}(f)\cdot \frac{H_{1}(f)}{H_{2}(f)}$ (5) $$

Let $\mathcal{W}_{1\rightarrow 2}=\frac{H_{2}(f)}{H_{1}(f)}$ (or $\mathcal{W}_{2\rightarrow 1}=\frac{H_{1}(f)}{H_{2}(f)}$), we can conclude that: 1) the receiver-to-receiver warping universally exists and 2) is uniquely defined for any pair of receiver positions, 3) it relies on the 3D space and is independent on audio source.

Proposition 2: If the more than one audio sources are placed in the 3D space, the receiver-to-receiver warping field existence is not guaranteed.

Proof: Assume $K$ ($K>1$) audio sources are placed in the 3D space, based on the superposition property in room acoustics [], one receiver recorded audio (e.g., $x_{1}(t)$) can be expressed as,

$$ $x_{1}(t)=s_{1}(t)\circledast h_{1,1}(t)+s_{2}(t)\circledast h_{2,1}(t)+\cdots+ s_{K}(t)\circledast h_{K,1}(t)$ (6) $$

where $h_{k,l}(t)$ indicates the RIR from the $k$-th source to the $l$-th receiver, By extending Eqn. (6) to $M$ receivers and further applying Fourier transform, we can get,

$$ $\left[\begin{array}[]{c}X_{1}(f)\\ X_{2}(f)\\ \cdots\\ X_{M}(f)\end{array}\right]=\left[\begin{array}[]{cccc}H_{1,1}(f)&H_{1,2}(f)& \cdots&H_{1,K}(f)\\ H_{2,1}(f)&H_{2,2}(f)&\cdots&H_{2,K}(f)\\ \cdots&\cdots&\cdots&\cdots\\ H_{M,1}(f)&H_{M,2}(f)&\cdots&H_{M,K}(f)\\ \end{array}\right]\cdot\left[\begin{array}[]{c}S_{1}(f)\\ S_{2}(f)\\ \cdots\\ S_{K}(f)\end{array}\right]$ (7) $$

For brevity, we can rewrite Eqn. (7) as $X=H\cdot S$. Since we have no knowledge of audio sources $S$, we can treat Eqn. (7) as multivariate polynomial linear function for audio sources $S$. To ensure a unique solution for $S$, two conditions have to be satisfied: first, the determinant of the coefficient matrix $H$ must be non-zero, det($H$) $\neq 0$. Second, the rank of coefficient matrix $H$ must be equal to the rank of the augmented matrix $[H|X]$, rank($H$) $=$ rank($[H|X]$). That is to say, special receiver placement is required to deterministically represent audio sources by receivers. Moreover, even if we can deterministically represent audio source by receivers, $S=H^{-1}\cdot X$, we can hardly predict one receiver’s spatial acoustic effects by warping from another single receiver because one receiver’s spatial acoustic effects, in multiple audio sources case, usually depend on multiple other receivers. We empirically show one such example in Appendix A.

### 3.3 LTI Receiver-to-Receiver Warping Field Physical Principle

We present three room acoustics physical principles [, ] that will guide *SPEAR* design.

Principle 1, Globality: Unlike a normal RGB image just captures a localized area, a receiver recorded spatial audio relates to the whole 3D space. Originating from the source position, the spatial audio propagates in a complex way that incorporates reflection, diffraction and absorption before reaching to the receiver position, resulting in the interaction with almost the whole 3D space before reaching to the receiver. Consequently, the final receiver recorded audio is influenced by the whole 3D space.

Principle 2, Order Awareness: The Order Awareness principle states that *SPEAR* should account for the specific order of the input two receivers. In essence, the learned warping field varies when the order of the two receivers is swapped. This can be readily proved by Eqn. (5), since $\mathcal{W}_{1\rightarrow 2}=\frac{H_{2}(f)}{H_{1}(f)}$, $\mathcal{W}_{2\rightarrow 1}=\frac{H_{1}(f)}{H_{2}(f)}$, $\mathcal{W}_{1\rightarrow 2}\neq\mathcal{W}_{2\rightarrow 1}$.

Principle 3, Audio-Content Agnostic: This principle asserts that the receiver-to-receiver warping field is an inherent characteristic of the 3D space, affected by neither the presence or absence of audio within the 3D space nor the specific class of audio.

Figure: Figure 2: Two challenges in *SPEAR* learning: Position-Sensitivity and Irregularity. The position-sensitivity is represented by much lower structural similarity index (SSIM) of two neighboring-step warping fields than the two RGB images (sub-fig. C). The warping field irregularity is represented by both warping field visualization in frequency domain (real part) and much higher sample entropy score than regular sine wave (and just half of random waveform) (sub-fig. D).
Refer to caption: x2.png

### 3.4 Position-Sensitivity and Irregularity of Receiver-to-Receiver Neural Warping Field

The complex behavior of spatial audio propagation in an enclosed 3D space often results in significantly different spatial acoustic effects even for audio recorded at neighboring positions. This position-sensitivity becomes even more pronounced in the receiver-to-receiver warping field, where even a small receiver position can lead to substantial warping field variation. As is shown in Fig. 2, we compare the visual differences and warping field variations caused by small receiver position change. Using the 3D space Office 4 from the Replica dataset [] and the SoundSpaces 2.0 simulator [], we place a stationary audio source and a reference receiver. Next, we ask the robot carrying a pinhole camera and a receiver to walk straightforward with step size $0.3$ m in the vicinity of the reference receiver. At each target receiver position (crimson color in Fig. 2), we capture RGB images and compute the corresponding warping fields. We then adopt structural similarity index (SSIM) [] to measure the visual and warping field differences between two neighboring positions. The results, shown in Fig. 2, clearly indicate that a 0.3-meter position change results in a significantly more pronounced warping field variation (much lower SSIM score) compared to the RGB images. To show warping field irregularity, we visualize the $5$-th step warping field real part in frequency domain in Fig. 2 D, from which we can see the warping field is highly irregular and thus exhibits higher sample entropy score [](^1^11The higher of sample entropy score, the more irregular is the series. To highlight warping field irregularity, we compare its sample entropy score with totally regular sine wave and random wave.) than regular sine wave.

### 3.5 SPEAR Neural Network Introduction

Figure: Figure 3: *SPEAR* network visualization.
Refer to caption: x3.png

The way we design *SPEAR* neural network is guided by Sec. 3.3 and Sec. 3.4. Specifically, *SPEAR* takes as input two receivers’ positions (one reference position and the other target position) and outputs the corresponding warping field that warps the spatial acoustic effects at the reference position to the target position (Audio-Content Agnostic principle applies). We build *SPEAR* on top of Transformer architecture [] and predict the warping field in frequency domain, in which the warping field is jointly represented by a real and imaginary series. Predicting the warping field in frequency domain results in both faster processing and better generation quality, as shown in the ablation experiment 4.3.
To accommodate the Globality principle and tackle the position-sensitivity challenge, we construct a learnable grid feature spanning to the whole 3D space horizontally, each cell of the grid thus corresponds to one particular physical position in the 3D space.
The two input positions’ features are extracted from the grid feature by bilinear interpolation.
Concatenating the two interpolated features in order (Order-Awareness principle satisfied) gets the input position-pair’s representation. Each token then combines the input positions’ representation and token index position encoding as the initial token representation.
After 12 transformer blocks, the final learned token representation is further fed a prediction head to predict their corresponding real/imaginary warping field. We visualize the neural architecture in Fig. 3.

Train and Inference.
To ensure the audio source covers the whole frequency range of the warping field, we choose Sine Sweep audio with frequency range [0-16] kHz. As only one audio source presents in the 3D space, we can obtain the ground truth warping field by dividing the target position received audio by source position received audio (see Eqn. (5)). We thus jointly train *SPEAR* with group truth warping field and two receivers’ recorded audio. The original warping field length we predict is 32 k. Due to the warping field symmetry (DFT conjugate symmetry), we just predict half warping field which are 16 k points. For ground truth warping field supervision, we combine both L1 and L2 loss. For the two receivers recorded audio supervision, we adopt spectral convergence loss []. More detailed discussion is in supplementary material.

## 4 Experiment

### 4.1 Experiment Configuration

Dataset. We evaluate *SPEAR* on three datasets across different domains.

1. Synthetic Dataset. We adopt Pyroomacoustics [] to simulate a large shoebox-like 3D space of size $[5m\times 3m\times 4m]$. All the receivers are placed on the same height, the audio source is placed at position $[2m,2m,2m]$. We simulate 3000 receivers positions for training and another 300 receivers for test, by ensuring the reference-target receiver pair position in the test set is significantly different from those in training set.

2. Photo-realistic Data. To show our model is capable of predicting the warp field in a more complicated and photo-realistic environment, we employ Replica Office 0 and Office 4 3D space [] to simulate the spatial audio. In Office 0 and Office 4, we simulate 4000 receivers and 8000 receivers for training, respectively. Both scenes have 500 receiver positions for test.

3. Real-World Dataset. We further test on the real-world classroom 3D space [] of size $(9~{}m\times 8~{}m\times 3.5~{}m)$. The audio source is placed on the platform in the classroom. A grid of $[10\times 13]$ receivers are placed on the on ground with spatial resolution $0.5~{}m$. 106 of the audio positions are used for training, and the rest 24 positions are reserved for test.

Figure: Figure 4: Learned warping field visualization on synthetic dataset (A) and real-world dataset (B).
Refer to caption: x4.png

Evaluation Metrics. To quantitatively evaluate the warping field, we adopt: 1. MSE, mean square error between ground truth and *SPEAR* predicted warping field in frequency domain (average between real and imaginary part). 2. SDR (signal-to-distortion ratio). Following [], we report SDR to assess the fidelity of predicted warping field. 3. PSNR (Peak Signal-to-Noise Ratio []) and 4. SSIM (structural similarity index measure []) to quantify the quality of the predicted warping field. We also introduce human-perceptual metric to provide insight into human perceptual quality of the predicted warping field. Specifically, we choose five speeches from VCTK [] dataset and warp them to the target position with the learned warping field, then compute the 5. PSESQ (perceptual evaluation of speech quality []) score for the warped speech and ground truth recorded speech.

**Table 1: Quantitative result on three datasets. MSE ($\downarrow$), SDR ($\uparrow$), PSNR ($\uparrow$), SSIM ($\uparrow$), PSEQ ($\uparrow$).**
| Method | Synthetic Data | Photo-Realistic Data | Real-World Data |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SDR | MSE | PSNR | SSIM | PSEQ | SDR | MSE | PSNR | SSIM | PSEQ | SDR | MSE | PSNR | SSIM | PSEQ |  |
| LinInterp | $-$0.37 | 1.40 | 14.58 | 0.85 | 1.65 | $-$0.94 | 1.44 | \cellcolortopcolor14.71 | 0.63 | 2.16 | $-$1.02 | 1.20 | \cellcolortopcolor 14.20 | 0.49 | 1.27 |
| NNeigh | $-$2.71 | 2.44 | 15.21 | 0.83 | 1.62 | $-$2.87 | 2.22 | 12.13 | 0.64 | 1.89 | $-$3.78 | 2.27 | 12.13 | 0.44 | 1.30 |
| NAF [] | 0.13 | 1.24 | 13.08 | 0.86 | 1.77 | 0.07 | 1.13 | 14.21 | 0.73 | 1.92 | $-$0.24 | 1.37 | 13.01 | 0.33 | 1.30 |
| *SPEAR* | \cellcolortopcolor1.50 | \cellcolortopcolor0.92 | \cellcolortopcolor15.81 | \cellcolortopcolor0.87 | \cellcolortopcolor2.47 | \cellcolortopcolor0.66 | \cellcolortopcolor1.03 | 14.57 | \cellcolortopcolor0.75 | \cellcolortopcolor2.18 | \cellcolortopcolor$-$0.24 | \cellcolortopcolor1.04 | 14.07 | \cellcolortopcolor0.75 | \cellcolortopcolor1.45 |

Comparison Methods Given the novel problem setting of *SPEAR*, currently there is no existing method that directly applies to our problem. Existing RIR-based spatial acoustic effects modelling methods [, , ] vary significantly in the way they exploit RIR data and the amount of prior room scene acoustic properties knowledge to train their model, we thus find it difficult to modify them to fit our setting. For meaningful comparison, we compare *SPEAR* with one source-to-receiver neural acoustic field learning method (NAF []) and two other interpolation methods.

1. NAF []. NAF learns continuous source-to-receiver RIR field by assuming access to massive RIR data. We modify it to accept two positions as input and output the warping field in frequency domain.

2. LinInterp: Neighboring Linear-Interpolation. For each receiver-pair in test set, we retrieve top-25 warping field data from the training set whose position are closest to the receiver pair (the position distance is defined by the the sum of reference position shift and target position shift). Then we average the 25 warping field to get the linear-interpolated warping field.

3. NNeigh: Nearest Neighbor method searches for the warp field, where the reference and target receiver positions are closest to the input test position pair. This is equivalent to retrieving the top-1 closest warping field from the training set.

### 4.2 Experimental Result

The quantitative result on the three datasets is given in Table 1, from which we can see that *SPEAR* outperforms all the three comparing methods by a large margin. Among all the five metrics, *SPEAR* maintains as the best-performing method (except for one PSNR metric). Moreover, *SPEAR* outperforms the second-best method NAF [] significantly on metrics like SDR and PESQ.

We also provide qualitative comparison of the predicted warping field in Fig. 4, in which we visualize the warping field (real-part) on Synthetic dataset (A) and real-world dataset (B) obtained by all methods. From this figure, we can clearly observe that 1) while the ground truth warping field exhibits high irregularity (see Sec. 3.4), *SPEAR* is capable of learning the irregularity pattern from the input position pair; 2) The two other methods (NAF [] and LinInterp) failed to tackle the irregularity challenge. NAF [] tends to predict close warping field values. The higher of the frequency, the easier it tends to predict the same value (see Fig. 4 A, the left and right part comparison of NAF).

### 4.3 Ablations

Figure: Figure 5: Ablation Study on noise interference (A), reference-target receiver distance (B) and Prediction in frequency domain and time domain.
Refer to caption: x5.png

We run all ablation studies on Synthetic dataset created by Pyroomacoustics [].

Warping Field Sensitivity to Noise. We add two ambient Gaussian noises (measured by SNR, 20 dB and 30 dB) to test model’s performance under noise interference. As is shown in Fig. 5 A, we can observe that while all three comparing methods have seen performance drop (increased MSE metric) as more noise is involved, *SPEAR* maintains as the best-performing method and outperforms the other two methods (NAF and LinInterp) by a large margin under all noise interference.

Warping Field Accuracy w.r.t. Reference-Target Receiver Distance. We further test *SPEAR*’s capability in predicting spatial acoustic effects for spatially distance target receivers. To this end, we compute models’ performance variation w.r.t. reference-target receivers’ distance change ($0.5~{}m-3.0~{}m$) and show the result in Fig. 5 B. We can observe from this figure that: 1) The increased reference-target receiver’s distance leads to performance drop (increasing MSE) for all methods, which is expected as the spatial acoustic effects drastically change when the receiver position gets changed dramatically. 2) *SPEAR* still outperforms the other comparing methods, showing its advantage in predicting warping field for further target receivers.

Warping Field Prediction in Frequency V.S. Time domain. In *SPEAR*, we propose to predict warping field in frequency domain for computation efficiency concern. Out of the computation efficiency concern, we further want to figure out the performance with predicting in time domain. As depicted in Fig. 5 C, we can observe that predicting warping field in time domain leads to significant performance drop as the model tends to predict all-zero warping field.

Audio-Content Agnostic Verification. Since all models are trained with Sine Sweep audio, we want to know if the Audio-Content Agnostic principle truly satisfies. To this end, we evaluate *SPEAR* with another three audios: engine, siren and human vocalization. As is shown in Table 2, we can verify that *SPEAR* is agnostic to audio content and can be applied to arbitrary audio class.

More experimental result and discussion are provided in supplementary material.

## 5 Conclusion and Limitation Discussions

We introduce a novel receiver-to-receiver spatial acoustic effects prediction framework that can be trained in a data much more readily accessible way. We theoretically prove the existence and universality of such warping field if there is only one audio source. When there are multiple sources, we can apply source separation method [] first to separate the sources first. There are two limitations that remain to be resolved in the future. The first is that we still require dense sampling to get reasonably good performance. Second, we assume all receivers’ position lies on the same horizontal plane. More in-depth investigation is needed to remove this constraint and allow receiver placement in the whole 3D space.

## Appendix

## Appendix A Receiver-to-Receiver Warping Field Existence Discussion

Figure: Figure I: Warping field visualization in frequency domain (real part). Top: warping field with two audio sources Engine and Footstep two audio sources. Bottom: warping field with the two audio sources at the same positions but the Footstep sources is replaced by Telephone Ring.
Refer to caption: extracted/5670699/imgs/2-source-warp-field.png

In this section, we empirically show that the receiver-to-receiver neural warping field may not exist if more than one audio sources present in the 3D space. To this end, we depend on Pyroomacoustics [] simulator to simulate a shoe-box like 3D space of shape $[5\times 5\times 5]$ meters. Two audio sources are placed at coordinate A $[1,1,1]$ meters and B $[4,4,4]$ meters, respectively. Two receivers are accordingly placed at coordinate $[1,1,2]$ meters and $[4,4,3]$ meters respectively. In the fist simulation, we place Engine audio at source position A and Footstep audio at source position B. In the second simulation, we just replace the Footstep audio at position B with Telephone Ring. With the two receivers recorded audio, we depend on Eqn. (5) to calculate the warping field from the reference receiver at $[1,1,2]$ to the target receiver at $[4,4,3]$. The computed two warping fields are shown Fig. I, we clearly see that the two warping fields are significantly different. We thus can conclude that the receiver-to-receiver neural warping field is no longer solely dependent on receiver positions (so the Audio-Content Agnostic principle does not apply). When more than audio sources present in the 3D space, the warping field proposed in this work may not exist.

## Appendix B Discussion on Acoustic Neural Warping Field Visualization

The acoustic neural field is represented in frequency domain, each of which in our case contains a real and imaginary one-dimensional data vector. In the main paper, we just visualize the real part due to the space limit. Here, we provide another five real/imaginary warping fields visualization in Fig. II. From this figure, we can clearly see that our proposed framework *SPEAR* is capable of handling the warping field irregularity property.

Figure: Figure II: Visualization of real and imaginary component of the ground truth warp field in the synthetic dataset.
Refer to caption: x6.png

## Appendix C Failure Case Visualization

During the experiment process, we find all methods inevitably give failure case warping field predictions. We visualize part of some failure cases predicted by *SPEAR* on both synthetic data (Fig. III), photo-realistic and real-world dataset (Fig. IV). As we discussed in the main paper, the position-sensitivity and irregularity pose challenges in the warping field prediction. We hope these failure cases will attract more investigation into this research problem.

Figure: Figure III: Examples of failure cases of SPEAR model on synthetic data. Regions with significant miss-match between the ground truth and the predicted warping field pattern are shaded in blue.
Refer to caption: x7.png

Figure: Figure IV: Failure case visualization of *SPEAR* model on both Photo-realistic and Real-world Dataset.
Refer to caption: x8.png

## Appendix D Training Detail Presentation

### D.1 Ground Truth Warping Field Acquirement

We obtain the ground truth by dividing the target receiver audio by the reference receiver audio in frequency domain. This division operator sometimes leads to NaN value or abnormally large value ($>100$, when the denominator is close to zero) in the obtained warping field (see the ground truth warping field visualization in Fig. V), resulting in the difficulty of accurately warping field learning. To handle this dilemma, we make two adjustments: first, replace NaN value with zero so that the whole neural network is trainable with the warping field prediction loss, which was NaN without the replacement.
Second, clipping all warping field values to lie within $[-10,10]$. The reason for the clip operation is two-fold: the abnormally large value easily allures the whole neural network to be trapped in predicting those abnormally large values, thus ignoring predicting the warping field with normal values; we further empirically verify in Fig. V that clip operation gives subtle difference in the warped target audio.

Figure: Figure V: Visualization of the effect of preprocessing Warping field on the generated audio waveform. In plot A C, we show the warping field and waveform of the target position without warping-field preprocessing. In plot B D, we show the pre-processed warping field and the waveform at the target position after applying the preprocessed warping field. Plot E shows the difference between the two waveforms at the target position.
Refer to caption: x9.png

### D.2 Training Configuration Presentation

We adopt AdamW optimizer [] for training on all datasets. On the synthetic dataset, the model requires approximately 10000 epochs to converge, which takes around 11 hours on a single A10 GPU. We set the learning rate of the learnable grid feature to 1e-5, and the rest learnable parameters’ learning rates to 1e-4. Using a smaller learning rate for the grid features improves the model training stability. Since the model predictions rely solely on the grid feature extracted, changes in grid feature can result in significant differences in model prediction. Therefore, setting a lower learning rate for the grid features prevents the model prediction from changing abruptly, and thus improves training stability.

Though our model has larger parameter size, the inference time is smaller than the NAF baseline. As shown in Tab. I, our model has more than ten times larger parameter size than the NAF model, but inference speed is around ten times faster than the NAF model.

**Table I: Model param. and Inference Time comparison. The inference time is the average of 1000 independent inferences with batch size 32 on a single A10 GPU.**
| Method | Inf. Time | Param. Num. |
| --- | --- | --- |
| NAF [] | 0.13 s | 1.61 M |
| *SPEAR* | 0.0182 s | 27.26 M |

### D.3 Data Sampling Strategy

For all three types of datasets, we construct train and test datasets by first splitting the receiver positions into two disjoint sets. The reference and target receiver positions are sampled from the same receiver position set. This means that, during both the training and testing stages, the model will not be trained/tested to predict a warping field that warps audio from a receiver position in the training set to a position in the test set, or vice versa.

In the synthetic data generation, we arrange the receivers in an $80\times 40$ grid, with adjacent receivers spaced 0.05 meters apart. To create the test set, we select test samples so that no two test samples are adjacent in the grid. This interleaved sampling strategy ensures that each test receiver position is at least 0.05 meters away from any receiver position in the training set.

In the Photo-realistic data generation, we randomly sample 4500 and 8500 receiver positions on the two scenes’ floors and select a subset of 500 receiver positions from each set as the test set. Due to the existence of furniture and other obstacles presented in the room scene, we could not employ the grid-sampling strategy used in synthetic data generation, and could only randomly sample train and test receiver positions on the scene floor.

To train our model on the real dataset, which has a significantly smaller sampling density required by our model, we first simulate the real dataset scene using the Pyroomacoustic simulator [] and generate 10800 samples in the scene. The receiver positions are arranged in a $90\times 120$ grid, with adjacent receivers spaced 0.05 meters apart. 1000 test samples are selected from the total 10800 receiver positions. The test set sampling strategy is the same as the interleaved sampling strategy used in the synthetic data generation.

We pretrain our model on the simulated samples before fine-tuning on the real dataset. Receiver positions in the real dataset are also arranged in a grid structure, which allows us to use the interleaved sampling strategy to sample 24 test set receiver positions from the total 130 receiver positions.

### D.4 Effect of Sample Size on Model Training performance

In all three datasets, reference and target audios are densely sampled from the scene. In this section, we show the necessity to sample audio at high density by training our SPEAR model on the same synthetic scene with different sampling densities. We randomly select a subset of 1000 samples and 2000 samples from the whole 3000 synthetic training data samples and show their performance metrics. Tab. II shows the metric of the three models. The model performance drops significantly as the sampling density decreases. In addition, we visualize the predicted warping field of the three models in Fig.VI. Models trained with smaller sample sizes show significantly worse performance in higher frequency warping field prediction.

**Table II: Effect of different sampling density on model performance.**
| Sample Size | SDR | MSE | PSNR | SSIM |
| --- | --- | --- | --- | --- |
| 3000 | 1.50 | 0.92 | 15.81 | 0.87 |
| 2000 | 1.06 | 0.96 | 15.27 | 0.87 |
| 1000 | 0.24 | 1.15 | 14.26 | 0.85 |

Figure: Figure VI: Predictions of models trained with different sample size.
Refer to caption: x10.png

## Appendix E SPEAR Network Architecture

*SPEAR* network architecture is shown in Tab. III, and we also provide the code in the suppementary material.

**Table III: *SPEAR* Network Architecture Detail.**
| Layer Name | Filter Num | Output Size |
| --- | --- | --- |
| Model Input: 2 position 3d coordinate: [2, 3] |  |  |
| Grid Feature: concatenated 2 position feature: [1, 384] |  |  |
| Transformer Encoder Input: Initial Token Representation: [43, 384] |  |  |
| Transformer Layer 1 | head num = $8$, hidden dim = $384$ | [43, 384] |
| … | … | … |
| Transformer Layer 12 | head num = $8$, hidden dim = $384$ | [43, 384] |
| Prediction Head |  |  |
| Real part FC | FC, output_feat = 384 | [43, 384] |
| Imaginary part FC | FC, output_feat = 384 | [43, 384] |
| Flattern | Flattern real/imaginary token sequence. | [16512] |
|  | Construct complex sequence. |  |
| Prune | Cut the sequence to 16384 length | [16384] |
| Mirroring | Generate the full warping field by | [32768] |
|  | concatenating the predicted sequence |  |
|  | with its mirrored conjugate sequence |  |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Could the authors elaborate on potential methods to reduce the dense sampling requirement? Would techniques like data augmentation be feasible for this purpose?

2. Are there specific changes or enhancements that could allow SPEAR to handle multi-level environments with varying elevations?

3. For real-time applications, what optimizations could be implemented to further reduce inference time without sacrificing accuracy?

4. Could the authors provide a comparison with conventional RIR-based approaches, detailing the trade-offs in accuracy, efficiency, and data requirements?
soundness: 3
strengths: 1. The receiver-to-receiver formulation of spatial acoustics is innovative, providing a new paradigm in spatial audio modeling that does not rely on prior knowledge of acoustic properties.
2. Methodologically rigorous, supported by a blend of theoretical analysis and robust experimental results across diverse datasets (synthetic, photo-realistic, real-world).
3. The overall flow and structure of the paper are clear, with detailed explanations of each stage in the model’s development, including physical principles and architecture specifics.
4. SPEAR’s method addresses a gap in the field, making spatial acoustics modeling more accessible for real-world applications in complex environments.
summary: The paper introduces SPEAR, a novel neural warping field model designed to predict spatial acoustic effects in a 3D environment with a single stationary audio source. Unlike traditional source-to-receiver models requiring prior knowledge of room acoustics, SPEAR operates from a receiver-to-receiver perspective, allowing it to predict how audio would sound at different spatial positions using only discrete audio recordings at various receiver positions. This framework is trained using synthetic, photo-realistic, and real-world datasets, demonstrating significant flexibility and generalizability across different environments. The paper's contributions include a new problem formulation, a theoretically supported neural architecture guided by three physical principles, and comprehensive experimentation showing SPEAR's accuracy and efficiency over baseline methods.
weaknesses: 1. Sampling Density Requirement: A dense sampling of receiver positions is currently required for SPEAR to achieve optimal accuracy. This requirement may limit its scalability in highly variable environments.

2. Positioning Constraint: SPEAR assumes all receiver positions lie on the same horizontal plane, which could restrict applications in multi-level or irregular environments. Addressing this limitation would extend the model’s utility.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: * SPEAR learns the propagation of sound from one receiver (reference) to the second receiver (target). So to map all potential (reference, target) positions in the room is also combinatorically complex (as in the RIR case). Thus:
  * How much more efficient this method is to the previous ones (RIR-based / NAF)?
  * Please explain more clearly the benefits (or tradeoffs) between SPEAR and source-to-receiver modeling methods. In other words, why does the latter require prior space acoustic properties knowledge and SPEAR does not? (this is not clear in both the introduction and not in “related work” section)
  * How dependent is the method on the specific source signal? If the signal is narrowband (single frequency), then the method would not be able to estimate other frequencies.
* There are two very significant claims in lines 162-165:
  * Warping transform is multiplicative in the frequency domain
  * The acoustic neural warping field is independent on audio content
While you refer to the appendix for proof and discussion, I would suggest adding some coarse and intuitive explanation to why that is the case.
* Equation (5) - there are cases where the transfer function H may be zero for some frequencies. Then the relations in equation (5) would not hold. How do you handle these cases? (in the experiments you mention that you use a Sine Sweep signal but this is not clear at this stage of the paper)
* Equation (7) is a generalization of eq. (5): thus the determinant condition applies to equation (5) as well, but it is not mentioned.
* In your SPEAR training you fix the position of the source. If we would like to use SPEAR with a source that is located in a different position (and in a different room), we would have to train it again in the new setting, correct? If so, this should be mentioned clearly
* What applications could benefit from this method (especially compared to alternatives)?
soundness: 2
strengths: * The method can be applied relatively easily since it does not require a lot of knowledge about the environment
  * It is relatively original since most other methods require more information about the environment or a more complex recording setup
  * Quality is somewhat unclear (see comments below)
  * Clarity and significance could be improved (see comments below)
summary: The authors propose a novel method for estimating the warping field that corresponds to sound propagation between two points inside a fixed environment. The method applies for a stationary source and two moving receivers (microphones). The receivers are synchronized, and thus the transfer function between the reference to the target receiver can be estimated from recordings in the frequency domain.
weaknesses: * The method estimates the ratio between two transfer functions and this is prone to ill conditioning - however, it is not fully addressed. In the experiments they simply use clipping and zeroing to handle such cases, but how this affects performance is not clear
* The strengths and weaknesses of the method are not sufficiently clear
* I’m finding some difficulties in understanding exactly what SPEAR is trying to learn. In line 304: “...we can obtain the ground truth warping field…” - if you can calculate it analytically and then train the model to predict it, what exactly does the model learn? Some kind of generalization to other frequencies? Interpolation to grid points that are not measured? In any case, this is not discussed in the experiments
* Experiments section (section 4.6)
  * There is too much emphasis on visual analysis of the warping field signal in the frequency domain (Figures 4, 6, 7, 8). It is very difficult 
to understand actual performance from these graphs. In all cases it looks like the estimated warping field is very different from the ground truth
  * Fig 5 - what is the meaning of the MSE values? Again, difficult to understand something about performance from reading absolute MSE values (i.e., how bad/good is an MSE of 1.06?)
  * Table 3 - was the metric measured for the same signal? It should be noted that depending on the frequency content of the signal, estimating the warping field is limited
* It is not clear how to use the method given some very important parameters:
  * How close should the receiver be? Is this frequency dependent?
  * What source signal to use? 
  *How many grid points should be sampled in the environment in order to estimate the warping field of the entire environment? Is this even possible?

**Minor comments**
* Introduction → contribution 3: “We demonstrate SPEAR superiority…” - superiority compared to what?
* Lines 132-139: the relation to “Time-series Prediction” is not very clear. Please explain more clearly what is the relation between SPEAR and the type of networks you mentioned.
* Equation (2) - is p_1 and p_2 the same as p_r and p_t ? if so, please be consistent with notation
* Lines 308-309: “For ground truth warping field supervision, we combine both L1 and L2 loss.” - can you please provide an explicit equation for the loss? Is there a weighting parameter between the L1 and L2 losses? Why did you incorporate both L1 and L2?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Several questions related to the weaknesses mentioned above:

1.	What is the motivation for learning the RTF where both receivers can change positions while source position is fixed (in test time maybe we would like the source to change places)?
2.	What is the relation to previous works on RTF estimation [2-4]?
3.	What is the key contribution over [5]?

Additional minor questions: 

4.	Why is the Fourier transform used instead of STFT representation?
5.	Line 151 - the meaning of this notation $\mathcal{F}\leftarrow(\mathcal{A},\mathcal{P})$ is unclear.
6.	There is inconsistency in the notation, switching from $p_1,p_2$ to $p_r,p_t$ and the same for the wrapping field.
7.	Proposition 2 contains an over-general statement (“existence is not guaranteed”) and the proof is vague. In general, the identifiability of the mixing matrix in (7) was investigated under the field of independent component analysis (ICA), and there are certain conditions for which the matrix is identifiable (full rank matrix and at most one Gaussian source).  The wrapping field can be defined as a vector of length K that contains the wrapping field for each individual source. 
8. Line 294: "The two input positions’ features are extracted from the grid feature by bilinear interpolation" - please clarify how the bilinear interpolation is performed. 
9. What is the motivation for using a transformer? What does the attention between patches is expected to learn? 	
10. The proposed method is not necessarily more realistic compared to baselines. The required data is similar to collecting massive RIR data, since RIR can be extracted when the source signal is known. 
11.	Fig 4., why NN baseline is not presented?
12.	The figures order does not follow the order they appear in the text -  Fig. 6 should come before Fig. 5.
13.	Warping Field Sensitivity to Noise - is the noise added during training?   What happens in higher noise levels?
14.	Missing details regarding experiments – what is the reverberation time? What is the signals length? 
15.	Why Fig. 5.a. does not contain comparison to NN? 
16.	Appendix D is empty
17.	Are there any insights regarding the characteristics of the failure cases?
18.	Table 3 – It is unclear why the feature dimension is 384, and what is 43 in the initial token representation. Why is there a pruning step at the output?
soundness: 3
strengths: The paper presents a novel viewpoint that learns the wrapping field that connects two receivers using a new transfomer-based model. A comprehensive experimental study is conducted with both simulated and real-world data, and different aspects of the proposed method are examined. The paper is clearly written and contains meaningful illustrations that clarify its core ideas.
summary: The paper presents a method for estimating the wrapping field in an enclosed acoustic environment, which is the relative transfer function between two receivers, when given the position of the two receivers. This method is meant to replace direct room impulse response estimation which relies on prior space acoustic properties and complex computations or requires massive amount of RIRs as a supervision. The method is shown to outperform baselines on both simulated and real-world data.
weaknesses: The usefulness of the proposed method is not well motivated. On the one hand, the wrapping field corresponds to a fixed source position, but in reality it is more informative to consider a source that can change locations. On the other hand, the space of all possible wrapping fields seems to be unnecessarily large, since it is defined by two receiver locations, whereas for RIR estimation the mapping is a function a single receiver only (and a source position). This is also evident from the vast amount of training samples that are required for training the model. Note that from these same measurements one can extract the RIR if the emitted source signal is known (which should be the case since the training recordings are performed in a controlled manner). 
Note that this idea of utilizing the wrapping field already exists in the literature for two decades. It is known as the Relative Transfer Function (RTF). A plethora of methods have been proposed to robustly estimate the RTF from measured signals (see a summary on [1] Section IV. C. 3). More close to the current paper, it was already proposed in previous works to generate RTFs based on source-receiver locations [2,3,4]. This relevant literature should be referred to in the paper, and it should be made clear what is the difference between these works and the current contribution.
In addition, the paper is very similar to [5], where it seems that the main difference is that the current paper deals with RTF estimation instead of RIR estimation, thus requiring an additional emitting sound at a fixed position. It should be clarified what is the main contribution of the current work compared to [5]. 

[1]  Gannot, S., Vincent, E., Markovich-Golan, S., & Ozerov, A. (2017). A consolidated perspective on multimicrophone speech enhancement and source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(4), 692-730.

[2] Wang, Z., Vincent, E., & Yan, Y. (2017). Relative transfer function inverse regression from low dimensional manifold. arXiv preprint arXiv:1710.09091.‏

[3] Wang, Z., Li, J., Yan, Y., & Vincent, E. (2018, April). Semi-supervised learning with deep neural networks for relative transfer function inverse regression. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 191-195). IEEE.‏

[4] Bianco, M. J., Gannot, S., Fernandez-Grande, E., & Gerstoft, P. (2021). Semi-supervised source localization in reverberant environments with deep generative modeling. IEEE Access, 9, 84956-84970.‏

[5] He, Y., Cherian, A., Wichern, G., & Markham, A. (2024, January). Deep Neural Room Acoustics Primitive. In Forty-first International Conference on Machine Learning

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: A few minor comments based on the weaknesses mentioned above.

- For the claim “to be more favorable to real-world scenarios” to be convincing, the method would need to be validated in cases where the source signal is not a sine sweep, but a more readily accessible sound like clapping (which is easy to generate and record at the receivers but difficult to acquire the original source). Furthermore, the performance under different receiver position sampling policies would need to be more rigorously validated to see how much space is covered by the receiver positions for training. In this respect, the choice of dataset in this paper could benefit from further diversification:

    - Pyroomacoustics was synthesized based on the image-source method in a shoebox room as stated in the paper

    - Sound Spaces 2.0 is also based on ray-based simulation and the office subset of Replica used by the authors is a single-room

    - MeshRIR is a real measurement, but the structure of the room is also a shoebox

    A dataset that can verify performance for non-trivial cases may be a multi-room dataset (where there could be no direct acoustic path, e.g., 'line of sight'), such as Replica's Apartment subset. In this case, other factors than the Reference-Target Distance reported in this paper may be important for sampling receiver positions. For example, if the source is in the living room of an apartment, how much detail should the training receiver be sampled to ensure performance?

- Typos/misleading phrases
    - L182, L205: The part where the author expresses "according to room acoustics (Savioja & Svensson, 2015)" seems to assume an LTI system, but since not all room acoustics assume LTI, the expressions seem to need clarification.
    - L370 PSESQ $\to$ PESQ
    - Figure 8 comes before Figure 6 and 7.
    - Wrong references to "Fig. B" (L456) and "Fig. C" (L464)
soundness: 2
strengths: One of the strengths of this paper is its value as an attempt to alleviate the difficulty of data acquisition for sound field synthesis. It also deserves recognition for achieving large computational efficiency gains within moderate memory efficiency.
summary: The authors point out that the problem of acoustic field estimation is characterized by the high cost of collecting prior information (about the physical properties such as geometries), and propose a relatively simple way to solve the problem: to estimate the receiver signal at another location from the receiver signal. They present their theoretical assumptions and backgrounds for their methodology and demonstrate its effectiveness on simulated and real recorded datasets.
weaknesses: For me, the justification for the main contribution of this study remains unclear. To point out my thoughts on the main contributions listed by the authors at the end of Section 1:

1. Please clarify the reasoning behind the claim that the receiver-to-receiver framework requires less prior information than source-to-receiver (e.g. NAF).

    - At the end of the second paragraph of Section 1, the authors explain their motivation as collecting RIR data is exceedingly difficult in real scenarios, and their receiver-recorded audio is more readily accessible. What I don't understand about this argument is that they ended up using a sine sweep source for training, so in what way is this more efficient than training with RIR?

    - Let's say that the training data is not necessarily a sine sweep. One of the cases where a receiver-to-receiver framework can be beneficial, as the paper suggests, is when we don't know about the source signal. I'm curious to hear your thoughts on whether cases like this are prevalent in real-world scenarios. In my understanding, as there should be a receiver recording to train SPEAR in a certain room, one should first play the source signal to acquire the responses at each receiver point, meaning that we already know the source signal.

2. The statements and proofs of the propositions in Section 3.2 seem to lack the rigor to be presented as major contributions.

    - What the authors describe as the claim and proof in the text for Proposition 1 is a direct consequence of Rayleigh's reciprocity theorem. Aside from the rigors in the proof of the “unique existence of warping given any pair of receivers”, there is something I don't understand about what the authors conclude in the last sentence of the proof: what does the claim “independent of the audio source” mean? Considering diffraction, the warping field should be variant depending on the source frequencies, isn’t it?
 
    - Similarly, there are several technical misleading statements throughout the paper that could be used to claim theoretical contributions, and the explanation of acoustic propagation is still underdeveloped. One example of this is the claim in Principle 1 in Section 3.3 that 'the entire 3D space is involved in the receiver recording', which could be misleading to those unfamiliar with audio. It would be better to claim instead that 'because the sound is generally more reflective than light, even signals reflected from occluded areas can be affected, and because of their lower velocity, the effect is more pronounced'.

3. To claim to have revealed the superiority of SPEAR, the baseline selection is weak, and the ablation study did not sufficiently reveal the strengths and weaknesses of the proposed methodology.

    - At the very least, INRAS [1] should be included, and it would be nice to see other baselines such as AV-NeRF [2], DiffRIR [3], etc.

    - In the context of synthesizing acoustic warping fields for moving sources with a fixed receiver (which shares systematic similarities with this paper’s problem statement), WaveNet-based architectures are often used [4,5], or even MLPs [6] to estimate the frequency domain warping fields. How does the Transformer architecture bring advantages over these other architectures?

    - I wonder if re-training is essential for comparing performance with RIR-based source-to-receiver methodologies (including NAF). Even with keeping those source-to-receiver models as-is, we can estimate $H_{p_r}$ and $H_{p_t}$ directly, and it seems natural to be able to obtain $\mathcal W_{p_r \to p_t}=H_{p_t}/H_{p_r}$. Is there a reason to retrain the NAF nonetheless? Given such commutative nature of LTI systems, how is receiver-to-receiver learning systematically different from source-to-receiver learning, as one could readily get the impulse response for each of the receivers and then deconvolve one into the other?

[1] Su, K., Chen, M., & Shlizerman, E. (2022). Inras: Implicit neural representation for audio scenes. Advances in Neural Information Processing Systems, 35, 8144-8158.
[2] Liang, S., Huang, C., Tian, Y., Kumar, A., & Xu, C. (2023). Av-nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Processing Systems, 36, 37472-37490.
[3] Wang, M. L., Sawata, R., Clarke, S., Gao, R., Wu, S., & Wu, J. (2024). Hearing Anything Anywhere. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 11790-11799).
[4] Richard, A., Markovic, D., Gebru, I. D., Krenn, S., Butler, G. A., Torre, F., & Sheikh, Y. (2021). Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations.
[5] Leng, Y., Chen, Z., Guo, J., Liu, H., Chen, J., Tan, X., ... & Liu, T. Y. (2022). Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances in Neural Information Processing Systems, 35, 23689-23700.
[6] Lee, J. W., & Lee, K. (2023, June). Neural fourier shift for binaural speech rendering. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.

Meta-review: This paper received contrasting reviews, indeed, 3 out of 4 reviews leaned more to negative evaluations (rating 5), while 1 was extremely positive (max rating, 10). After rebuttal and extensive discussion, the 2 of the former evaluations downgraded the rating to clearly negative (rating 3), one remained on 5, while the latter rating was decreased to slightly positive (rating 6).

This result was obtained after extensive discussion between the authors and the reviewers, and also the positive reviewer (rev. y6xA) significantly re-evaluated his/her judgment after the comments and discussion raised by the other reviewers.

In conclusion, for the above reasons, this paper cannot be considered acceptable to ICLR 2025.


The work deals with the estimation of the spatial acoustic effects in a 3D space for any receiver position when a stationary source emits sound. Not requiring neither source position nor 3D space acoustic properties, the proposed method SPEAR simply uses two microphones to actively record the spatial audio independently at discrete positions, and produces in output a warping field warping the recorded audio on reference position to target position and comparing it with real recorder audio. In short, SPEAR aims at acoustically characterizing the 3D space from a receiver-to-receiver perspective. 

Major good points relate to the originality of the work and the fact of not using much information for the task it addresses (differently from former works).
Weak points are several, regarding the clarity of the methodology in some parts, unclear contributions of the work, weak demonstration of good performance and several issues concerning the experimental results, not fully clear motivations, the need of better comparison with the state of the art. 

The authors discussed extensively wth reviewers, who acknowledged to have clarified some of these issues, but in the end they were not fully satisfied of the answer authors provided.

Decision: Reject