SoundNeRirF: Receiver-to-Receiver Sound Neural Room Impulse Response Field
Authors: Yuhang He
Jia-Xing Zhong
Zhuangzhuang Dai
Niki Trigoni
Andrew Markham
Venue: Submitted to ICLR 2023 (2023)
Keywords: Sound Neural Rendering Field
Sound Prediction
Representation Learning
Receiver-to-Receiver Modelling

Abstract: We present SoundNeRirF, a framework that learns a continuous receiver-to-receiver neural room impulse response field~(r2r-RIR) to help robot efficiently predict the sound to be heard at novel locations. It represents a room acoustic scene as a continuous 6D function, whose input is a reference receiver's 3D position and a target receiver's 3D position, and whose outputs are an inverse room impulse response~(inverse-RIR) and a forward room impulse response~(forward-RIR) that jointly project the sound from the reference position to the target position. SoundNeRirF requires knowledge of neither sound source (e.g. location and number of sound sources) nor room acoustic properties~(e.g. room size, geometry, materials). Instead, it merely depends on a sparse set of sound receivers' positions, as well as the recorded sound at each position. We instantiate the continuous 6D function as multi-layer perceptrons~(MLP), so it is fully differentiable and continuous at any spatial position. SoundNeRirF is encouraged, during the training stage, to implicitly encode the interaction between sound sources, receivers and room acoustic properties by minimizing the discrepancy between the predicted sound and the truly heard sound at the target position. During inference, the sound at a novel position is predicted by giving a reference position and the corresponding reference sound. Extensive experiments on both synthetic and real-world datasets show SoundNeRirF is capable of predicting high-fidelity and audio-realistic sound that fully captures room reverberation characteristics, significantly outperforming existing methods in terms of accuracy and efficiency.

Under review as a conference paper at ICLR 2023
SOUNDNERIRF: RECEIVER-TO-RECEIE SOUND
NEURAL ROOM IMPULSE RESPONSE FIELD
Anonymous authors Paper under double-blind review
ABSTRACT
We present SoundNeRirF,, a framework that learns continuous receiver-to-receiver neural room impulse response field (r2r-RIR) to help robot efficiently predict the sound to be heard at novel locations. It represents room acoustic scene as a continuous 6D function, whose input is reference receiver's 3D position and target receiver's 3D position, and whose outputs are an inverse room impulse response (inverse-Ril RiR) and a forward room impulse response (forward-RI) ( that jointly project the sound from the reference position to the target position. Sound- NeRirF requires knowledge of neither sound source (e.g. location and number of sound sources) Instead, it merely depends on sparse set of sound receivers' positions, as well the recorded sound at each position. We instantiate the continuous 6D func- tion as multi-layer perceptrons (MLP), so it is fully differentiable and continuous at any spatial position. SoundNeRirF encouraged, during the training stage, to implicitly encode the interaction between sound sources, receivers and room acoustic properties by minimizing the discrepancy between the predicted sound and the truly heard sound at the target position. During inference, the sound at novel position is predicted by giving a reference position and the corresponding reference sound. Extensive experiments show SoundNeRirF is capable of predicting high-fidelity and audio-realistic sound that fully captures room reverberation characteristics, significantly outperforming existing methods in terms of accuracy and efficiency.
acoustic properties (e.g. room size, geometry, materials).
both synthetic and real-world datasets
INTRODUCTION
Room acoustic [3, 50, 1, 27] aims accurately capturing the sound propagation process in a room environment, so to determine the received sound characterizing reverberation at any spatial position. Robots and other intelligent agents (e.g. voice assistants) can exploit such sound for a variety of tasks, including navigation [9, 44, 29]. reconstruction [62, 8] and audio-dependent simultancous localization and mapping (SLAM) [16]. As a complementary sensor to vision, sound perception exhibits strengths in scenarios involving dim acoustic can be treated as linear time-invariant system (LTI). Thus, the underlying sound propagation problem is to precisely derive a room impulse response (RIR) a transfer function of time and other acoustic variables that measures the behaviour sound takes from a sound source to a receiver. The sound recorded a particular position can thus be obtained by convolving the sound at the source position with the RIR. We call such RIR
lighting and occlusion. In most cases, room
RIR (s2r-RIR).
Accurately deriving s2r-RIR, however, a difficult task. The challenge three-fold: 1) high computational cost: traditional methods [3. 50, 1, 27, 41, 26, 4, 22] undergo rigorous mathematical derivation and extensive measurements to compute s2r-RIR, by treating sound as either rays [50, 1, 27] or waveforms [3, 41]. The corresponding computation complexity is proportional to the room geometric layout complexity and the number of sources. 2) non-scalable: the whole computation process needs to be re-executed once the source/receiver location changes slightly, or the room layout has altered (e.g. furmiture movement). 3) too strong assumptions: the assumption that the source/receiver location and room acoustic properties are well- and known in advance is too strong to be held in real scenarios. The sound source location is mostly unknown in scenarios,, and localizing sound sources is extremely difficult task [20, 23, 6, 19].

---

Under review as a conference paper at ICLR 2023
In this work, we propose SoundNeRirF,, a treceiver-to--eceiin Sound Neural Room impulse response Field for efficiently predicting what sounds will be heard at arbitrary positions. SoundNeRirF requires knowledge of neither sound sources (e.g. source number and position) but instead represents a room acoustic scene through a sparse set of 3D spatial positions that robots have explored, well as the sound recorded by the robot at each position. A robot equipped with receiver can easily collect massive such datasets by simply moving around the room and recording its position and received sound at each step. SoundNeRirF represents room acoustic scene by a continuous 6D function, which takes two receivers' 3D positions (one reference and one target position) as input, and outputs two room impulse responses that jointly project the sound from the reference position to the target position. SoundNeRirF thus learns receiver-to-receive RIR). SoundNeRirF is reinforced to implicitly encode the interaction between sound sources and receivers, in addition to room acoustic properties by minimizing the error between predicted and the truly recorded sounds, because sound recorded at any position is naturally a result of such an interaction. By instantiating the 6D function multi-layer perceptrons (MLPs), SoundNeRirF is differentiable and continuous for any spatial position, and can be optimized with gradient descent. Figure visualizes SoundNeRirF's motivation.
room acoustic properties'
RIR (r2r-
Specifically, SoundNeRirF splits r2r-RIR into two parts: an inverse- tieers-RRRRae for the reference position and a forward-RIR for the target position. We further introduce receiver virtual position settings and a physical constraint strategy that guides SoundNeRirF to explicitly learn direct-path, specular- reflection and fate-reverberation that commonly exist in room acoustics. SoundNeRirF has numerous applications in robotics, including immersive audio/video game experience in augmented real- ity (AR) [24, 57], sound pre-hearing without actually reaching to the location [5], and improving robot localization (by utilizing the predicted sound and truly heard sound).
In summary, we make four main contributions: First, we propose SoundNeRirF that learns r2r-RIR neural field from a sparse set of receiver recordings. Second, SoundNeRirF requires knowledge of neither sound source nor room acoustic properties, but instead depends on more accessible data that can be collected by robot walking around in the room environment. Third, SoundNeRirF directly predicts sound raw waveform. It disentangles sound prediction and 12r-RIR Icarning, exhibiting strong generalization capability in predicting previously unheard sound (see Experiment Sec.). Lastly, we release all datasets collected in either synthetic and real-world scenario to the public to foster more research.
2 RELATED WORK
Room Acoustics Modelling There are two main ways to model room acoustics: wave-based mod- elling [3, 41, 26, 4] and geometry-based modelling (aka geometrical acoustics) [50]. The wave-based modelling utilizes sound wave nature model sound wave propagation, whereas geometry-baae based modelling treats sound propagation as optic rays. Typical geometry-basee modelling methods include ray tracing [27], image source method (ISM [1]), beam tracing [17] and acoustic radiosity [22, 36). The main goal of room acoustics is to characterize the room reverberation effect, which consists of three main components: direct-path, specular-refiectionn and late-reverberation. SoundNeRirF explicitly models the three components to learn R2r-RIR.
Neural Room Impulse Response (RIR) Some recent work [46, 55, 45, 12, 30, 49] proposed to learn room impulse response (RIR) with deep neural networks. However, they all assume massive nource-to-receiver RIRs (s2r-RIR) are available to train the model. Unlike these methods, Sound- NeRirF learns implicit R2r-RIR from a set of robot recorded sounds at different positions, which are easiler to collect with much less constraints.
Neural Radiance Field (NeRF) has received lots of attention in recent years, especially in computer vision community [33, 21, 59, 52, 61]. They model static or dynamic visual scenes by optimiz- ing an implicit neural radiance field in order to render photo-realistic novel views. Some recent work [60, 13, 58] extends to neural radiance field to 3D point cloud [60], image-text domain [58] and robotics [13, 53, 35, 28] for robot localization, position and scene representation. A. Luo et. al. [30] propose to learn implicit neural acoustic fields from source-receiver pairs model how nearby physical enviromment affect reverberation. However, their method requires the presence of massive
rrom acoustic properties relate to any factor that may affect sound propagation, including room size, geometric structure, surface absorption/scatterng coefficient, etc.
2

---

Under review as a conference paper at ICLR 2023
W3 :
P1+ -mmphowllll
F(P)
<
P2t mmllmmmnn
SoundNeRirF
P3 + ?
p4+ Mnnippall
www + P3
Pn + -ww.Appeeor
Record at Various Positions
B. Receiver has Position and Sound
C. Optimize SoundNeRirF
Infer Novel
Figure 1: SoundNeRirF motivation: in geometric acoustic room, multiple receivers (or robot walks different positions) at various positions are recording the sound emitted from multiple sound sources (A). Each receiver records the sound and its spatial position receiver-to-recetee neural room impulse response field (C), so as to predict the sound at novel position, given a reference sound and a reference position (D).
(B). SoundNeRirF learns
s2r-RIR as well. Our proposed SoundNeRirF shares the same idea with NeRF that learns an implicit receiver-to-receiver acoustic room impulse response field.
Sound Synthesis Inferring sound waveform for a novel location partially relates to sound synthe- sis [38, 14, 64, 48, 15, 54, 11, 42]. WaveNet [38] learns to predict future sound waveform based on previously heard sound waveform. WaveGAN [14] and GANSynth [15] adopt generative adversarial network (GAN [18) to learn to generate sound, Specifically, A. Richard et. al. [48] propose to infer binaural sound waveforms from mono-channel input waveform.
SOUNDNERIRF: RECEIVER-TO-RECENE NEURAL ROOM IMPULSE RESPONSE FIELD
3.1 PROBLEM DEFINITION
In geometric acoustic room, at least static sound source is constantly emitting sounds. N sound receivers at various positions (or as a robot moves to different positions) are recording the acoustic environment, {(W, P) = { (w,, p;) 1. W R P E R Each sound w; is time series of the sound raw waveform (of length T) and position is the receiver's s coordinate [x; V/. zi]. Our target learn an implicit receiver-to- coneiver-to--eeetie re-reveverr neural room impulse response field (SoundNeRirF) FF from (W, P), that is capable of efficiently inferring the sound tù, that would be heard at novel position (target position) pt. Note that pt is previously unexplored, and we are given reference sound w,, from explored location Pr.
r8g = ((W.P)(P. Pr. wr), (16r, Pt) & (W,P), (w..pr) E (W,P)
(1)
In this work, both sound source and receiver are omni-directional and monoaural, so we do not have to consider their directivity and inter-channel difference (as discussed in [20, 48]). We also assume all receivers record the same sound content (e.g. telephone ring) but each has encoded unique room reverberation effect based its position, so the sound at arbitrary position be predicted by the sound from any reference position, if the room reverberation change between them can be modelled.
3.2 SOUNDNERIRF FRAMEWORK INTRODUCTION
SoundNeRirF represents the receiver-to-receiver neural room impulse response field as 6D function h, whose input is the combination of one reference receiver position p, = [xy, yr. žy] and one target receiver position p: [xt, Va, 2,], and whose output is inverse-RIR inv that mainly depends on reference position pr and a forward-RIR h for that mainly depends on target position Pr: ji fo h(pr,Pt) Pt). havv and fo jointly project the sound at the reference position w, to the target position ny via 1D convolution operation,
w w. + h for
where @ indicates ID convolution operation in time domain. Both inv and for one-dimensional impulse response in the time domain, and we omit their time indexes for succinct representation. The motivation behind learning two separate RIRs is to reduce the reference-target position arbitrarity
3

---

Under review as a conference paper at ICLR 2023
22, 1
for tif
C = (M,, wi))
virtually sources
2nd-ordar
invense. RIR
forward-Re RRR
A
for
IIV h
h
for
h for
h
Ist-order
MLP
MLP
P
Position Encoding
Position Encoding
2
p
Pr
2
Pr
Pt
Pt
3
P4
3 p
(W+rP+)
H2: SoundNeRiri sets virtual receiver positions
SoundNeRirF Pipeline Illastration
B. Viriual Receiver Setting
Figure 2: SoundNeRirF pipeline (left) and virtual receiver position setting motivation illustration.
impact and explicitly exploit room acoustic RIR property: we first project the reference sound to a virtual implicit sound source field with inverse-Rin RIR (h""V, mainly depends reference position) and then reproject it from the virtual implicit sound source field to the target sound with forward- RIR for mainly depends on target position). By explicitly setting physical constraints I and oo (Sec. 3.4), SoundNeRirF learns more robust r2r-RIR representation. In this work, we instantiate both in and for as two identical multi-layer perceptrons (MLP, specifically 6 layers with 512 neurons in each layer). so they are fully diffcrentiable and can be optimized by gradient descent algorithm. They are also continuous at any spatial position we can infer the sound for any position,
h iuv MLPin (pr), foro -MLPforpp))
(3)
MLP representation has been successfully used for acoustic tasks such speech enhancement [37] and audio equalization [40]. We extract the last MLP layer representation as the r2r-RIR (see Fig. 2). Please note that l and foror learning independent of the sound content, predicting previously unheard sound (sound emitted from the same source but not trained with, see Experiment). Hitherto we have vanilla SoundNeRirF with raw receiver position input learn neural R2R-RIR field using an MLP. We further modify it by adding extra virtual receivers and setting physical constraints to reinforce SoundNeRirF learn more representative r2r-RIR neural field.
use them for
3.3 SETTING VIRTUAL RECEIVER POSITIONS
To explicitly model the direct-path, pvecular-reflection and late-reverberation effect that commonly exist in room acoustic RIR, we manipulate each original input 3D position (which holds both the reference and target receiver position) and consecutively place them at two virtual further locations (in our case, multiply the original position by different coefficients to ensure the receiver is virtually set at further positions). The resulting three positions are leveraged to explicitly model the direct-path RIR ha (original position), specullr-reflection RIR h2 and the late-reverberation RIR hys respectively. The final RIR is obtained by adding three separate RIRs together: ha(O1 p) + hg (a2 P)) + hı(axy p). and three separate RIRs go through the same MLP. Please note that the virtual receiver position setting is adopted for both the reference position and the target position. As a result, la in Eqn. can be rewritten as,
hinv iinv thinv biuuN h
MLPiny (aLPi"((a1pP)) (ar MLPiny MLPinY(ay (appr), hi MLPinv (as
for can obtained similar formulation Eqn. 4. a1, 022, 03 are hyperparameters setting the receiver to further virtual position (In our case, they are empirically chosen, @1 1, 01 2, a3 3). The motivation of such design is based on sound propagating as optic rays [1, 27]: the longer the path sound signal travels, the greater the time delay and energy decay of the RIR there are in general. We implicitly approximate such room acoustic RIR nature caused by sound travelling path length difference by virtually setting the receiver the further location. Similar approximation has been used by a traditional Imaged Sourced Method (ISM [1]), in which mirrors the sound source against wall to get virtual sound source that has different distance to the receiver. The mirroring
4

---

Under review as a conference paper at ICLR 2023
process can be executed one multiple times to obtain different order virtual sources. Since we have no knowledge of the sound sources, we instead manipulate the receiver location to model the same RIR nature (See Fig. 2 right). Following the practice in [43, 63, 56, 331, add position encoding to encode the 3D position into high-dimensional representation before feeding it to neural network.
3.4 PHYSICAL CONSTRAINT ON R2R-RIR
To encourage SoundNeRirF to learn representative r2r-RIR, we set two constraints MLP-leamed RIR: First, time-delay der constraint for direct path ha, specular-reflection ha and late- reverberation hy. Second, reflect the increased amplitude attenu- ation (energy decay) caused by longer travelling distance, we add monotonicity constraints both inverse-RIe RIR and forward-RIR.
0.10
0.05
0.00
0.05
inverse-
For the MLP-leamed inverse-RIR nn (specifically hTy, hiv, h!mr) and forward- RIR A for (specifically ffor hefer h(a)) " in Eqn. 4, associate each individual RIR with learnable time shift pa- rameter (N ginv t/my for inverse-RIR, ffor ffor ,ffor for forward- RIR). By restricting the time shift parameters in ascending or- der (that is, & < 's S. timy, tar S forr S tfar) we set con- straints the RIRs in Eqn. 4 reflect different travelled distances of sound. Moreover, to reflect the greater energy decay (attenuated amplitude) caused by longer travelling distance, we add mono- tonicity constraint to each learned RIR separately. For example, for the lav the two physical constraints can be expressed as,
0.10
100
200
0.10
0.05
0.00
forward-RL RIR
-0.10
1000
200
Figure Learned iuv and for
h juv (t) = ht inv (t jay
Id (t) = (ha jav
inv
inv (t-1))
(5)
where t is the RIR index in the time domain. The other five RIRs can be obtained in similar way. (.) operation ensures monotonicity. In inverse-RIn iinv we use increasing monotonicity which means max(-), while in forward-RIR. for Examples of typical learned inverse-RIR and forward-RIR on our synthetic dataset are shown in Fig. 3.
decreasing monotonicity o-)) min(-)-
3.5 TRAINING AND INFERENCE
The SoundNeRirF pipeline is shown in Fig. 2. It takes input pair of reference and target receiver positions. Two identical MLPs with layer-wise communication [34, 6] are used to learn f and of in parallel. Position encoding is exploited to embed both the real and virtual positions to higher dimensions before feeding to MLP. The "nv and foor are jointly optimized by minimizing the discrepancy between the predicted sound ú; and the ground truth sound Wy. For the loss function, we combine mean squared error in both time and time-frequency domains,
L = ||w - 12 [STFT(()) - STFT
6
where STFT(-) indicates the short-time Fourier transform. We explicitly use the mean squared loss in the frequency domain because merely resorting to mean squared loss in the time domain easily leads to audio quality issues and distorted signals [48]. During inference, theoretically we can leverage any robot traversed position and recorded sound (reference sound) to infer the sound (target sound) at an arbitrary novel location.
4 EXPERIMENT
Dataset We run experiments three datasets:
1. Synthetic Dataset Single large room with 3D source/receiver positions. With the flexibility of arbitrarily setting source/receiver number or positions, we can test the necessity of each components of SoundNeRirF framework (ablation study). To this end, we adopt Pyrooma- coustics [51] to simulate a large shoebox-lie room of size [50m x 50m x 30m]. Seven seed sounds are collected from riph-right free website freesound. org: piano, cat-meowing, foot step, phone-Be iiee-riis ringing, taby-cryime dog-barking and people-talking Each seed sound is 4 seconds long and with the sampling rate 16k Hz. To reflect the impact of
5

---

Under review as a conference paper at ICLR 2023
Syuthetic Keul-Worid (left Robot, right: Flocrplaa) Choss-RRom Data (lefi: scene, right: tnjeccuryy)
Figure 4: Three datasets comparison. A. We can freely put arbitrary sources/receivers in the synthesized shoebox- environment. B. We use Turtlebot to collect sound in lab and corridors. C. Robot explore a multiple-room scene collect sound various positions. The trajectory path on the topdown map evolving from blue to red color indicates the exploration chronological order.
10.
601
0.4-
50-
TF-MSE
IF-MSE
0.3-
0.34
0.42
TF
3.0
0.14
0.09
0.08
0.08
0.
0.03
0,03
0.03
0.0g
100
300
Mediun
For
6
Receiver Deusity
Reference-Tuaee Distance
Sound Souree Number
WarpNet
TCN
WaveNet
SoundNeRirF
Figure 5: TF-MSE variation change w.r.t receiver density (left), reference-target distance (middle) and sound source number (right). Please note we re-scaled the vertical axis for the left/right figure for better visualization and comparison.
sound source numbers, we simulate seven identical rooms to cover sound source numbers from one to seven. In each room, we first put the corresponding sound sources at random positions, then randomly set 500 receivers at various locations (by guaranteeing at least 0.5 m distance to any sound source) to record the sound simultaneously. In each room, keep all the sound sources and receivers' positions unchanged but replace the original seed sounds with new ones. helps test all methods' generalization capability in predicting unheard sound. The resulting 500 recorded sounds each room are split into 300/100/100 for train/test/eval, respectively.
Real-World Dataset Lab/Corridor environment. We use a Turtlebot robot equipped with a sound receiver (hosted by Raspberry Pi). Vicon system [32] is utilized for localizing both speaker and receiver with a high precision (millimetre-leell precision). The data-collection area consists of a cluttered lab room and two adjacent corridors. Up in the lab area with fixed positions. The Turtlebot moves to different positions to record sound. -eyying
omni-directional speakers put
We collected two sub-datasets: the first one involves one sound source (emitting first and then p piano. baby-cry ing is for training model, piano is for unheard sound test), while the second one has two sound sources (one emitting cat-meow wwwng and the other emitting dog-bark ing at the same time). In total, we collected 120 receiver positions for one and two sound sources separately, with 90/30 for train/test split.
Cross-Room Dataset large area with multiple rooms. To test the performance in room scenes that contain multiple rooms, and complex room layouts (e.g. wall, sofa and other furniture), we further run experiments on large-scale acoustic-realistic room environment platform SoundSpaces- 2.0 [10]. SoundSpaces-2.0 supports Matterport3D indoor environment [7], it synthesizes high- fidelity sound (both binaural and monoaural) for any given source and receiver position. The average scene size is larger than 100m². In total, have collected 5000 sound recordings at different positions throughout the room scenes (each room with 1000 recordings). We randomly split it into 700/300 for train/test, respectively. We train the individual model for each room separately. Each room has five sound sources that distribute to whole area, all sound sources emitting telephone-ring sound.
The comparison of the three datasets is in Table 8. More discussion about the three datasets' creation/mottvaation is given in Appendix A. Figure 4 shows the three datasets comparison. All datasets will be publicly released to benefit the research community.

---

Under review as a conference paper at ICLR 2023
Table Overall Result on Synthetic Data
Table 2: Unheard Sound Result on Synth. Data
Methods
T-MSE
TF-MSE
CDPM
Methods
T-MSE
TF-MSE
CDPM
LinInterp
0.97
0.61
1.29
Linlnterp
0.97
0.61
1.39
NNeigh
1.12
0.78
1.41
NNeigh
1.12
0.78
1.44
WarpNet [48]
1.17
0.42
1.28
WarpNet [48]
1.38
0.66
1.12
WaveNet [38]
0.27
0,08
1,41
WaveNet [38]
1.23
0.58
1.41
TCN [2]
0.43
0.14
1.45
TCN [2]
1.21
0.76
1.47
SoundNeRirF
0.19
0.03
0.43
SoundNeRirF
0.19
0.04
0.44
Evaluation Metrics 1) T-MSE: time domain mean squared error (T-MSE). It evaluates the difference between predicted sound and ground truth sound the time domain; 2) TF-MSE: time-firequee frequency mean squared error (TF-MSE). We first convert the ID sound waveform to 2D time-frequend" representation by short-time Fourier transform (STFT), and compute the mean squared error in the fime-freyuency domain; 3) CDPM [31, 64]: Contrastive learning-based deep perceptual audio metric. It incorporates human judgement in the loop, and is trained with a dataset carefully labelled by humans according their perceptual similarity. We use CDPM as an approximation of human subjective evaluation of two sounds similarity. The lower of each of the three metrics, the better of the performance.
Test Configuration For the synthetic dataset, we create 50 reference sounds (selected from the training set) that cover the whole room. The sound at a test position is predicted by each of the reference sounds separately. So each test position has 50 predictions. In addition to reporting the average metrics, we rank the 50 predictions according to their distance to the test position, and divide them into three distance-aware categories: close, medium and far. Reporting results under the three categories helps to understand different methods' capability in predicting sound from distant, farfield positions (see Fig. 5 middle). We randomly select 30/50 reference sounds from the training set for real-world/cross-ror room test, respectively.
Comparing Methods There prediction. We compare SoundNeRirF with five most relevant methods: two interpolation based methods and three neural network based methods (More detailed discussion is in Appendíx B):
existing methods that directly model receiver-2-receivee sound
1. LinInterp: neighboring inhear-nteerpolatinn For each test position, query its closest positions with known sound. Then we linearly interpolate the sound to be heard at the test position by merging the 5 neighboring sound recordings with per-position weight, and each weight of the sound at a known position is inversely proportional to its distance to the test position.
NNeigh: nearest neighbor. Without any leariing/iterpooatiinn process, we simply treat the nearest neighbor sound as the sound to be heard at any test (novel) position.
3. WaveNet [38]. The widely used WaveNet adapted to take reference receiver position, sound and target position input. It exploits dilated convolution to predict sound waveform.
4. TCN [2]. TCN adopts convolution and recurrent neural networks to model sequence2sequence transformation. We inject both the reference and target position to TCN reinforce TCN to learn to predict target sound.
WarpNet [48]. WarpNet learns a warping field that maps the monoaural audio to binaural audio. In our implementation, feed receiver position [x, y, to the warping module learn to warp the sound from the reference position to the target position.
Training Details We train all models with Pytorch [39] and Adam optimizer [25], with an initial learning rate 0.0005 which decays every 20 epochs. We train 100 epochs and train each model three times and report the mean value for reducing model initialization impact (standard deviations are all within 0.01, we do not report it for brevity report). All sounds have sampling rate 16 k Hz. The sound length for both synthetic and worldworld data 4 s, while for cross-room data is s. All sounds are normalized to |-1, 1] before feeding the neural network for train. SoundNeRirF network architecture given Appendix E.
4.1 EXPERIMENTAL RESULT ON SYNTHETIC DATASET
The result on synthetic dataset is shown in Table I and Table 2 (T-MSE: 10 10-3). We can see that SoundNeRirF achieves the best performance over all the five comparing methods. In unheard sound
7

---

Under review as a conference paper at ICLR 2023
Table One Sound Source Result on Real Data
Table 4: Two Sound Sources Result on Real Data
Methods
T-MSE
TF-MSE
CDPM
Methods
T-MSE
TF-MSE
CDPM
LinInterp
0.37
0.07
0.91
LinInterp
2.50
0.10
0.88
NNeigh
0.62
0.10
0.98
NNeigh
4.20
0.13
0.97
WarpNet [48]
0.27
0.06
0.70
WarpNet [48]
3.10
0.10
0.85
WaveNet [38]
0.30
0.01
0.89
WaveNet [38]
2.05
0.06
1.17
TCN [2]
0.60
0.02
0.70
TCN [2]
2.31
0.07
0.96
SoundNeRirF
0.20
0.01
0.70
SoundNeRirF
1.20
0.05
0.71
Table 5: Result Cross-Room Data
Table 6: Param Size/Inference Time
Methods
T-MSE
TF-MSE
CDPM
Inference Time
Methods
Params
15 CPU
Rasp.
LinInterp
3.40
0.59
3.10
NNeigh
4.50
0.77
4.89
WarpNet [48]
8.59 M
2,925
WaveNet [38]
1.90
2.657
9.377
WarpNet [48]
3.12
0.30
2.31
TCN [2]
8,57 M
2,991 s
WaveNet [38]
1.76
0.19
1.43
TCN [2]
2.04
0.26
1.68
SoundNeRirF
3.29
0.011 s
0.066
SoundNeRirF
1.64
0.10
1.07
test, the three learning comparing methods have observed remarkable performance drop (even worse than the two non-learning baselines), while SoundNeRirF maintains nearly the same performance and thus exhibits strong generalization capability in predicting previously unheard sound.
Receiver Density Discussion results reported in Table 1 and are obtained by models trained 300 receiver recordings. To figure out the impact receiver density (number receiver recordings), we train another models with much smaller densities (with 100, and 200 recordings respectively). The result is shown Fig. 5 left, from which we can see WarpNet [48] the most sensitive to receiver density. The reduced receiver density leads significant performance reduction. TCN [2] also suffers from low receiver density, and WaveNet [38] is less sensitive to receiver density than WarpNet and TCN. SoundNeRirF invariant receiver density change (TF-MSE stays as 0.03), which attests that SoundNeRirF is able to learn representative r2r-RIR with limited receiver recordings.
Reference-Target Distance Discussion. The TF-MSE variation w.r.t. reference-target distance is shown in Fig. middle, from which we can see that the performance of WarpNet and WaveNet gradually decrease as the reference-target distance increases. SoundNeRirF and TCN are insensitive to reference-targee distance, and SoundNeRirF stays as the best under all distance metrics.
Impact of Source Number. The performance w.r.t. sound source number is shown in Fig. 5 right, from which we can see the three comparison methods reduce in accuracy the number of sources increases, with WarpNet [48] suffering the most. SoundNeRirF remains the hest-performing method, owing its design in learning implicit neural r2r-RIR field.
4.2 ABLATION STUDY
Table 7: Ablation Study Result on Synth. Data
To test the necessity and effectiveness of each component of SoundNeRirF design, we test three SoundNeRirF variants: 1) SoundNeRirF with- out physical constraints (SoundNeRirF (Soun)))eRifFnnoPhyss).. which helps test the necessity of explicitly set- ting physical constraints; 2) SoundNeRirF with- explicitly modelling inverse- RIR and forward- RIR (SNeRirF IMLP, instead using one MLP to predict r2r- 22-RIRR: 3) SoundNeRirF without receiver virtual position setting (SNeRirF Vit). The result is given in Table (T-MSE: 10-3), from which we can see the necessity of each SoundNeRirF component. Removing each component inevitably leads inferior performance.
Methods
T-MSE
TF-MSE
CDPM
SNeRirF noPhys
0.32
0.05
SNeRirF_
0.35
0.05
0.57
SNeRirF Vi
0.34
0.05
0.60
SoundNeRirF
0.19
0.03
0.43
8

---

Under review as a conference paper at ICLR 2023
0.10
Category
Synthetic Data
Real- World Data
Cross- Room Datasct
6:5-
8.00
inversc-Ri RIR
forward-RIl RRR
Source Num.
1-7
1-2
5 s
-0.10
100
250
200
Sound Length
4
s
I Learned Cor2Cor RIR
Data Size
500 x
120
1000 5
0.10
Area Size
[50, 50, 30]
[10, 6, 90/30
2 100
Train/Test
300/100
700/300
0.00
Data Distribution
3D Simulator Synthetic
2D Plane
2D Plane Robot
-0.0
Data Collector
TurtleBot
inverse-RIR
forward-RIR
Data Source
Real
Synthetic
100
200
Learned Lab2Cor RIR
Table 8: Three Dataset Comparison
Figure 6: SoundNeRirF RIR Vis.
4.3 EXPERIMENTAI. RESULT ON REAL- WORLD DATASET
The result on real-world dataset is given in Table 3 (one source) and Table (two sources) (T-MSE: 10-t). We can learn that 1) while all methods have sharp performance drop when transiting from one sound source to two sound sources, SoundNeRirF largely mitigates the dilemma and always performs the best. Typical SoundNeRirF learned RIRs are shown in Fig. 6 and Fig. VI,V, VII in Appendix, with reference and target receiver in different positions (lab or corridor). We see that SoundNeRirF successfully characterizes the room acoustic and learns different r2r-RIRs for different positions. Table 9: TF-MSE (x10 Result on Real Data (one source, eeboy-cryin w.r.t Lab/Cor Categories.
Cross-Wall Wall Prediction Discussion We fur- ther divide the reference and target receivers
Cross-
in the test set into two main categories: those in the lab room and those in the corridor, then report the performance by differentiating the reference and target re- ceiver positions. We thus get four de- tailed results: Lab2Cor, Lab2Lab, Cor2Cor, and Cor2Lab (in reference-to-tuee to-target format, Lab2Cor indicates the reference receiver is in the lab room, and the target receiver is in a corridor). The result is shown in Table 9. We can see 1) SoundNeRirF outperforms all comparing methods in all Lab/Cor categories. 2) WaveNet shows advantage in Lab2Cor and Cor2Cor sound prediction, but performs poorly in the other two categories. More discussion is in Appendix Sec. C.I and Table II. 4.4 EXPERIMENTAL RESULT ON CROSS- ROOM DATASET
Methods
Lab2Lab Lab2Cor Cor2Cor Cor2Lab
WarpNet [48]
3.20
5.18
10.2
5.09
WaveNet [38]
1.72
0.13
0.16
1.77
TCN [2]
1.97
1.08
2.56
2.14
SoundNeRirF
1.57
0.08
0.07
1.62
The result cross-room data is given in Table 5 (T-MSE: 10-1), from which can that SoundNeRirF stays as the best-performing method, it outperforms existing methods by large- margin (especially the two non-learning baselines). thus shows SoundNeRirF generalizes well to large and complex room acoustic environment where various physical impediments, and room layout exist. Non-learning based methods lead the worst performance (huge performance drop), which shows received sounds largely vary w.r.t. position displacement in complex room environment.
Inference Time/Parameter Size result is in Table 6. The average inference time is computed by averaging 100 independent tests (in each test, we use one reference sound to predict one target sound) on two hardwares: 1.6 GHz Intel Core i5 CPU (single core) and Raspberry 4. For Raspberry device, covert models to ONNX-compatee cannot be converted to ONNX- compx--bmatible models). We can see SoundNeRirF has small parameter size (just larger than WaveNet), and is extremely efficient. We refer readers Appendix C, D for more discussion, including learned r2r-RIR visualization, extra results and noise discussion. Appendix H to potential application discussion. In sum, Rorn exhibits superiority in predicting sound to be heard at novel positions. It generalizes well for unheard sound prediction and requires much less training data than existing methods. It can also handle physical impediment.
models (WarpNet [48] model and TCN [2] model
CONCLUSION AND LIMITATION DISCUSSION
In this work, we have proposed a receiver-to-reeeeee to-receiver acoustic neural field for predicting the sound to be heard at novel positions, given the sound heard at known positions. We assumee the acoustic scene is static and sound sources position is fixed. Moreover, we just focus on mononaural sound. One future work is extend to binaural sound prediction.

---

Under review as a conference paper at ICLR 2023
REFERENCES
[1] Jont B. Allen and David A. Berkley. Image method for efficiently simulating small-room acoustics. In The Journal of the Acoustical Society of America, 1979.
[2] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolu- tional and recurrent networks for sequence modeling.
/003://8000//// 2018.
[3] Stefan Bilbao and Brian Hamilton. Wave-based room acoustics simulation: Explici/iiplliit finite volume modeling of viscothermal losses and frequency-depender boundaries. Journal of the Audio Engineering Society, 2017.
[4] D. Botteldoore. Finite-difference time-domain simulation of low-frequency room acoustic problems. Journal of the Acoustical Society of America, 1995.
[5] James Broderick, Jim Duggan, and Sam Redfern. The importance of spatial audio in modern games and virtual environments. IEEE Games, Entertainment, Media Conference (GEM), 2018.
[6] Yin Cao, Turab Iqbal, Qiuqiang Kong, Fengyan An, Wenwu Wang, and Mark D Plumbley. An improved event-independent network for polyphonic sound event localization and detection. In International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
[7] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. International Conference on 3D Vision (3DV), 2017.
[8] Boyuan Chen, Mia Chiquier, Hod Lipson, and Carl Vondrick. The boombox: Visual reconstruc- tion from acoustic vibrations. In Annual Conference on Robot Learning (CoRL), 2021.
[9] Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, and Kristen Grauman. Soundspaces: Audio-visual navigation in 3d environments. In European Conference Computer Vision (ECCV), 2020.
[10] Changan Chen, Carl Schissler, Sanchit Garg. Philip Kobernik, Alexander Clegg, Paul Calamia, Dhruv Batra, Philip W Robinson, and Kristen Grauman. Soundspaces 2.0: A simulation platform for visual-acoustic learning. arXiv, 2022.
[11] Samuel Clarke, Negin Heravi, Mark Rau, Ruohan Gao. Jiajun Wu, Doug James, and Jeannette Bohg. Diffimpact: Differentiable rendering and identification of impact sounds. In Annual Conference on Robot Learning (CoRL.), 2021.
[12] Enzo De Sena, Hacihabihoglu Hüseyin, Zoran Zoran, Cvetković, and Julius O. Smith. Efficient synthesis of room acoustics via scattering delay networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2015.
[13] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W. Taylor, and Joshua M. Susskind. Unconstrained scene generation with locally conditioned radiance fields. International Conference on Computer Vision (ICCV), 2021.
[14] Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Interna- tional Conference on Learning Representations (ICLR), 2019.
[15] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Guirajani, Chris Donahue, and Adam Roberts. GANSynth: Adversarial neural audio synthesis. In International Conference on Learning Representations (ICLR), 2019.
[16] Christine Evers, Alastair H. Moore, and Patrick A. Naylor. Acoustic simultaneous localization and mapping (a-slam) of moving microphone array and its surrounding speakers. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016.
[17] Thomas Funkhouser, Nicolas Tsingos, Ingrid Carlbom, Gary Elko, Mohan Sondhi, James West, Gopal Pingali, Patrick Min, and Addy Ngan. A beam tracing method for interactive architectural acoustics. Journal of the Acoustical Society of America, 2003.
[18] Ian Goodfellow, Jean Pouget-Abaee Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeruIPS), 2014.
Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
[19] Yuhang He and Andrew Markham. SoundDoA: Learn sound source direction of arrival and semantics from sound raw waveforms. In Interspeech, 2022.
10

---

Under review as a conference paper at ICLR 2023
[20] Yuhang He, Niki Trigoni, and Andrew Markham. SoundDet: Polyphonic moving sound event detection and localization from raw waveform. In International Conference on Machine Learning (ICML), 2021.
[21] Peter Hedman. Pratul P. Srinivasan, Ben Mildenhall. Jonathan T. Barron, and Paul Debevec. Bak- ing neural radiance fields for real-time view synthesis. International Conference on Computer Vision (ICCV), 2021.
[22] Murray Hodgson and Eva-Marie Nosal. Experimental evaluation of radiosity for room sound- field prediction. Journal of the Acoustical Society of America, 2006.
[23] Jinbo Hu, Yin Cao, Ming Wu, Qiuqiang Kong, Feiran Yang, Mark D. Plumbley, and Jun Yang. A track-wise ensemble event independent network for polyphonic sound event local- ization and detection. In IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), 2022.
[24] Yasuhide Hyodo, Chihiro Sugai, Junya Suzuki, Masafumi Takahashi, Masahiko Koizumi, Asako Tomura, Yuki Mitsufuji, and Yota Komoriya. Psychophysiological effect of immersive spatial audio experience enhanced using sound field synthesis. In International Conference on Affective Computing and Intelligent Interaction (ACII), 2021.
[25] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representation (ICLR), 2015.
[26] Mendel Kleiner, Bengtinge Dalenbäck, and Peter Svensson. Auralization-an overview. Journal of the Audio Engineering Society, 1993.
[27] Asbjørn Krokstad, S. Strøm, and Svein Sorsdal. Calculating the acoustical room response by the use of ray tracing technique. Journal of Sound and Vibration, 1968.
[28] Yunzhu Li, Shuang Li, Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. Conference on Robot Learning (CoRI.), 2021.
[29] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. In International Conference on Learning Representa- tion (ICLR), 2016.
[30] Andrew Luo, Yilun Du, Michael J Tarr, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Learning neural acoustic fields. arXiv preprint arXiv:2204.0065 2022.
[31] Pranay Manocha, Zeyu Jin, Richard Zhang, and Adam Finkelstein. CDPAM: Contrastive learning for perceptual audio similarity. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2021.
[32] Pierre Merriaux, Yohan Dupuis, Rémi Bouttcau, Pascal Vasseur, and Xavier Savatier. A study of vicon system positioning performance. Sensors, 2017.
[33] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision (ECCV), 2020.
[34] Ishan Misra, Abhinay Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2016.
[35] Arthur Moreau, Nathan Piasco, Dzmitry Tsishou, Bogdan Stanciulescu, and Arnaud de La Fortelle. Lens: Localization enhanced by nerf synthesis. In Conference on Robot Learning (CoRL), 2021.
[36] Eva-Marie Nosal, Murray Hodgson, and Ian Ashdown. Investigation of the validity of radiosity for sound-field prediction in cubic rooms. Journal of the Acoustical Society of America, 2004.
[37] Soha A. Nossier, Julie Wall, Mansour Moniri, Comelius Glackin, and Nigel Cannings. A comparative study of time and frequency domain approaches to deep learning based speech enhancement. In International Joint Conference on Neural Networks (IJCNN), 2020.
[38] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In arXiv: (tt.:v:660000499990 2016.

---

Under review as a conference paper at ICLR 2023
[39]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Viti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Sys- tems (NewrIPS), 2019.
[40] Giovanni Pepe, Leonardo Gabrielli, Stefano Squartini, and Luca Cattani. Designing audio equalization filters by deep neural networks. Applied Sciences, 2020.
[41] Andrzej Pietrzyk. Computer modeling of the sound field in small rooms. Journal of the Audio Engineering Society, 1998.
[42] Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: flow-based generative net- work for speech synthesis. In International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2019.
[43] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning (ICML), 2019.
[44] Pratyaksh Rao and Abhra Roy Chowdbury. Learning to listen and move: An implementation of audio-aware mobile robot navigation in complex indoor environment. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[45] Anton Ratnarajah, Zhenyu Tang, and Dinesh Manocha. TS-RIR: translated synthetic room impulse responses for speech augmentation. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2021.
[46] Anton Ratnarajah, Shi-Xiong Zhang Zhang, Meng Yu, Zhenyu Tang, Dinesh Manocha, and Dong Yu. Fast-RIR: Fast neural diffuse room impulse response generator. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2022.
[47] Anton Jeran Ratnarajah, Zhenyu Tang, and Dinesh Manocha. IR-GAN: Room impulse response generator for far-field speech recognition. Interspeech, 2021.
[48] Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Butler, Fernando de la Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In International Conference on Learning Representations (ICI.R), 2021.
[49] Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Esti- mating and parameterizing filters with deep networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022..
[50] Lauri Savioja and U. Peter Svensson. Overview of geometrical room acoustic modeling techniques. Journal of the Acoustical Society of America, 2015.
[51] Robin Scheibler, Eric Bezzam, and Dokmanić Ivan. Pyroomacoustics: A python package for audio room simulations and array processing algorithms. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2018
[52] Shih-Yang Su, Frank Yu, Michael Zollhoefer, and Helge Rhodin. A-NeRF: Articulated neural radiance fields for learning human shape, appearance, and pose. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[53] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. iMAP: Implicit mapping and posi- tioning in real-time. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[54] Ee-Leng Tan, Woon-Seng Gan, and Chiu-Hao Chen. Spatial sound reproduction using con- ventional and parametric loudspeakers. In Asia Pacific Signal and Information Processing Association Annual Summit and Conference. 2012.
[55] Z. Tang, L. Chen, B. Wu, D. Yu, and D. Manocha. Improving reverberant speech training using diffuse acoustic simulation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Uszkoreit, Llion Jones Jone, Aidan
N. Gomez, and Lukasz Kaiser. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
12

---

Under review as a conference paper at ICLR 2023
[57] Charles Verron, Mitsuko Aramaki, Richard immersive synthesizer for environmental sounds. IEEE Transactions on Audio, Speech, and Language Processing (TASLP), 2010.
and Grégory Pallone. A 3-d
[58] Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao. CLIP-NeRF: Text- and-image driven manipulation of neural radiance fields. IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[59] Hongyi Xu, Thiemo Alldieck. and Cristian Sminchisescu. H-NeRF: Neural radiance fields for rendering and temporal reconstruction of humans motion. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[60] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann. Point-NeRF: Point-based neural radiance fields. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[61] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. Pixel-NeRF: Text-and-image driven manipulation neural radiance fields. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[62] Chengxi Zhong Zhong, Yuyu Jia, David C. Jeong, Yao Guo, and Song Liu. Acousnet: A deep learning based approach to dynamic 3d holographic acoustic field generation from phased transducer array. In IEEE International Conference on Robotics and Automation (ICRA), 2022.
[63] Ellen D. Zhong Zhong, Tristan Bepler, Joey Davis Davis, and Bonnie Berger. Reconstruct- ing continuous distributions of 3d protein structure from cryo-em images. In International Conference on Learning Representations (ICLR), 2020.
[64] Jan Zuiderveld, Marco Federici, and Erik J Bekkers. Towards lightweight controllable audio synthesis with conditional implicit neural representations. Advances in Neural Information Processing Systems Workshop (NeurIPS Workshop), 2021.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX
A APPENDIX 1: MORE DISCUSSION ON DATASET
A.1 DISCUSSION ON SYNTHETIC DATASET
In the synthetic dataset, Pyroomacoustics [51] simulate shoebox-like like room environment of size [50m x 50m x 30m]. The reason for creating a room of such size is two-fold:
A large room intrinsically involves large reverberation difference. Sound recorded by re- ceivers at different positions thus varies dramatically, in terms of sound waveform amplitude and other room acoustic related properties. Simulating such a room better fits for testing algorithms' capability in modelling room acoustics.
2. The simulated room size is complementary to the real-world dataset environment roughly size [12m x 12m x 3m]. which is much smaller than simulated room and robot was collecting sound data within nearly plane (the axis pointing to sky, same). This contrast with the synthetic dataset in which the receivers used to collect the sound are randomly set along the r- y- and x- axes, filling in the whole room.
axis are nearly the
To accurately model sound propagation process, combine ray tracing [27] and image source method [I] to derive the sound recorded by any receiver. Combining ray tracing and image source method strikes good balance between recorded sound accuracy and computational cost. For the room simulation, we set the origin point [0, 0, 0] to be located at one corner, and all receivers' position coordinates are larger than O.
A.2 DISCUSSION ON REAL-WORLD WORLD DATA
Lab Area
2-
0
Corridor 2
Corridor
Figure I: Real-world data receiver positions visualization. Color red: positions for train; color blue: positions for test.
Bacalver
Camiter
A. Robot Setup
B. Robot Collecting Sound in Corridors
C. Floor-plan Map
Figure II: Real-world data collection setup.
14

---

Under review as a conference paper at ICLR 2023
In the real-world environment, we have collected 65 receiver recordings in the two adjacent corridors, and 55 receiver recordings in the lab environment. These receiver locations cover all navigable area in the lab and two corridors. The train/test split is randomly chosen and uniformly covers both the lab and two corridors. In the real-world dataset, the x- axis range is [-2.0m m, 4.0 (-6.0ml m, 6.0 ml. 2- range is (0.0m. m. 0.3 m]. The plane formed by -- and y- axes is the 2D floor-plan map as shown in the main paper.
4.0mm, ml, y- axis range is
The TurtleBot sound receiver positions are shown in Fig. and detailed setup is in Fig. II, in which we can observe that the positions TurtleBot has traversed uniformly cover both the lab area and two adjacent corridors. The receivers used to train (color red) and test (color test) also cover the lab area and two corridors.
A.3 DISCUSSION ON CROSS-ROOD RooM DATA
A. Room Scese gYvKGGZ5ekdp
Room ARNzlegoob
Figure III: Embodied robot collects sound in complex room environment (figure A) and multiple rooms (figure B). The left sub-figure each figure shows the robot exploration trajectory in the top-down view, the trajectory evolving from blue to red color indicates the exploration chronological order. The right fugffigure visualizes the 3D scene environment.
To test the performance in room scenes that contain multiple rooms, complex room layout (e.g. room geometric features, sofa, bed and other furniture), we further run experiments on large-scale acoustic-realistic room environment platform SoundSpaces-2.00 [10]. SoundSpaces-2.0 supports Matterport3D photo-realistic indoor environment [7], synthesizes high-fidelity sound (both binaural and monoaural) for any given source position and receiver position. In our setting, we leverage the embodied robot to efficiently explore 5 room scenes. The robot step size is 1.0 m and turn-angle 30% it contains three actions: move- f forward, turn-leff and turn-right ritt. At each time step, it records a mono-channel sound that is I second long sound. Each room contain five sound sources that across the whole area, both emitting tel welephone-riie -rroonneeiiinngne sound. Figure III shows two scene examples and embodied robot exploration trajectory, from which we can see Matterport 3D room scenes represent world large indoor room environment where cluttered physical impediments and complex room layout exist. In total, we have collected 5000 sound recordings at different positions throughout the 5 room scenes (each room with 1000 recordings). For the 1000 recordings in each room, we randomly split it into 700 for train and 300 for test, respectively. We adopt the same training strategy the all models, as we do for the other two datasets.
B APPENDIX 2: MORE DISCUSSION ON COMPARING METHODS
B.1 TWO NON-LEARNING BASELINES
In our experimental setting, we have a set of receivers' recordings that distribute a room area. We use these known recordings predict the sound to be heard novel positions. Two simple baselines just interpolating the sound at novel position from a subset of receiver recordings with known sound and positions. Here we introduce two interpolation based baselines:
neighboring ihhear-iterrollatinn (LinInterp): for each test position (novel position), we query its closest 5 positions with known sound. Then we linearly interpolate the sound be heard at the test position by merging the 5 neighboring sound recordings with per-position weight, each weight of the sound a known position is inversely proportional to its distance to the test position and all weights sums to after so soft ftmmaa x constraint.
nearest neighbor (NNeigh): without any learning or interpolation process, we simply treat the nearest neighbor sound as the sound to be heard at any test (novel) position. This baseline
15

---

Under review as a conference paper at ICLR 2023
Table I: WaveNet comparison with one SoundNeRirF variant (SoundNeRirF (MLP) on Synthetic Dataset over sound source number from to 7.
Methods
T-MSE (c)
TF-MSE (1)
CDPM (4)
Params.
WaveNet [38]
0.27++0.00
0.08+0.01
1,41+0001
1.90 M
SoundNeRirF IMLP
0.35:10.00
0.05:10.11
0.57:10.01
1.64 M
helps to test if deep neural network based methods truly learn meaningful representation to sound rendering.
From the results the three datasets, we can learn that 1) The two non-learning based baselines (LinInterp and NNeigh) lead to inferior performance than the other four deep neural network based methods (three comparing methods and our SoundNeRirF); 2) LinInterp performs better than NNeigh, which shows incorporating neighboring sounds can better capture the sound characteristic at a novel position. In summary, room sound rendering, it helps to better characterize room acoustics.
conclude that: explicit learning process is essential for accurate
B.2 THREE LEARNING- BAADD METHODS
There are existing methods that directly model receiver-to-reeee sound neural field. We thus compare SoundNeRirF with three relevant methods: WarpNet [48], TCN [2] and WaveNet [38]. WarpNet [48] synthesizes binaural audio from monoaural audio. It learns warping field that maps the monoaural audio to binaural audio. In our implementation, we feed receiver position [z: y, 2] to warping module to learn the warp field. TCN [2] adopts convolution and recurrent neural network to model sequence2sqqeence transform. WaveNet [38] adopts dilated convolution to predict future sound, based on previous heard sound. In our implementation of TCN [2] and WaveNet [38], we concatenate receiver position [z. 2] to the input sound waveform and feed their concatenation to neural network. Please note that normalization of the input receiver position [x. y, 2] is applied as well, as we do in SoundNeRirF.
The comparison of learnable parameter number among these methods are shown in Table We can see from table that SoundNeRirF parameter number is much smaller than those of TCN [2] and WarpNet [48]. SoundNeRirF parameter number larger than the parameter number of WaveNet [38], mainly due to the two paralleling MLPs for modelling inverse-RIR and forward-RIR respectively and layerwise feature communication strategy. We find that SoundNeRirF without learning reverse-RIR and forward-RIn -IR respectively. but instead use MLP learn one general RIR (which means just use 6-layer MLP to learn r2r-RIR, we call it SoundNeRirF. IMLP) achieves comparable and even better performance than WaveNet [38] the synthetic dataset. The result is shown in Table I, from which we can see SoundNeRirF 1MLP outperforms WaveNet [38] in terms of TF-MSE and CDPM metries, with a much smaller parameter number. It thus shows the advantage of SoundNeRirF in modelling receever-to-reeeiver sound neural field.
The average inference time of the comparing three methods and SoundNeRirF is given in Table 6. The average inference time is computed by averaging 100 independent tests (in each test, we use one reference sound to predict one target sound) on two hardwares: 1.6 GHz Intel Core i5 CPU (single core) and Raspberry Pi 4. For Raspberry Pi 4 device, we covert trained models to ONNX DONX-compatible models and run ONNX models on the device. We find that WarpNet [48] model and TCN [2] model cannot be converted to ONNX-compatible models successfully due to various issues, such as operator incompatibiiity and unsupported operators. Therefore, we only report the inference time of WaveNet [38] and SoundNeRirF. From Table 6, we can clearly see that SoundNeRirF's inference time is much shorter than the three comparing methods' inference time, with just 1/200 of the three comparing methods' inference time CPU device, 1/150 of WaveNet [38] inference time on Raspberry Pi (although SoundNeRirF has more trainable parameters than WaveNet). Therefore, it shows the high efficiency of Soird.
16

---

Under review as a conference paper at ICLR 2023
C APPENDIX 3: MORE EXPERIMENTAL RESULTS
C.I TWO NON-LEARNING BASELINES ON CROSS LAB-CORRIDOR EXPERIMENT
We note that the neighboring sound recordings collected for the two non-learning based baselines lie very close the test position. As result, the sound recordings used to interpolate and the sampled nearest neighbor sound spatially lie in the same local room environment with the sound to be predicted (e.g. both lie in the lab environment or corridor environment, see Fig. I). To better test the two non-learning baselines' capability in predicting sound of further distance, we specifically compartment all sound recordings with known positions into different room environments (e.g. lab and corridor), and just use sound recordings in one environment to predict the sound in another room environment. The result (under TF-MSE evaluation metrics) shown in Table II, from which we can that the performance of LinInterp and NNeigh deteriorates significantly when predicting sound in one room environment by sound recordings from another different room environment (the largest TF-MSE increase by Lab2Cor and Cor2Lab). It thus shows explicitly learning process is required to implicitly encoding room acoustic properties.
Table II: TF-MSE (x10- Result Real Data (one source, baby-crying) w.r.t Lab/Cor Cate- gories.
Methods
Lab2Lab Lab2Cor Cor2Cor Cor21ab
Linlnterp NNeigh
4.10
8.21
12.1
9.23
4.81
9.17
13.0
10.67
WarpNet [48]
3.20
5.18
10.2
5.09
WaveNet [38]
1.72
0.13
0.16
1.77
TCN [2]
1.97
1.08
2.56
2.14
SoundNeRirF
1.57
0.08
0.07
1.62
C.2 MORE QUALITATIVE RESULTS, LEARNED RIR VISUALIZATION
We first provide two learned r2r-RIR visualizations on MP3D dataset [7]. We provide one corridor-tor corridor (Cor2Cor, both reference and target position are in corridor) learned receiver-to-rrceiver RIR in Fig. IV. From this figure, we can clearly see that in large spatial area MP3D dataset [7] where the average area is larger than 100 m² the direct-path RIR, specular-reflection RIR and reverbbratannn RIR learn different time shift parameters. Such different time-shift parameters characterize room acoustics properties large spatial area where the direct-path sound arrives to a receiver position earlier than specular-reflection sound, specular sound arrives earilier than later-reverberation sound.
We also provide corridor-to-corride (Cor2Cor, both reference position and target position are in corridor) learned receiver-to-recein RIR in Fig. V, lab-to-corriddo (Lab2Cor, reference position in lab, target position are in corridor) learned receiver-to-rceeiver RIR in Fig. VI. lab-to-lab (Lab2Lab, both reference and target position are in lab) learned weceiver-to-receieee RIR in Fig. VII. From the three figures, we can see that in much smaller area in our real-world dataset, the learned time-shift difference between direct-path. rpecular-reflection and enter-reverberation is much smaller. Such phenomena is compatible with real-scenario room acoustics property which shows that in small area the arrival time for direct-path, specular-reflection and late-reverberation very close.
In summary, from such learned r2r-RIR visualizations, we can conclude that SoundNeRirF learned RIR capable of characterizing 3D room scene acoustically, from the receiver-to-recevver modelling perspective.
C.3 MORE ABLATION STUDY
In the main paper, we have tested three SoundNeRirF variants: 1) SoundNeRirF without physical constraints (SoundNeRirF _nnFhys),,,.. which helps test the necessity of explicitly setting physical constraints; 2) SoundNeRirF without explicitly modelling inverse-RIR and forward-RIR (SNeR- irF IMLP, instead using one MLP to predict r2r-RIR. It achives slightly better performance than WaveNet [38]); 3) SoundNeRirF without receiver virtual position setting (SNeRirF no The
17

---

Under review as a conference paper at ICLR 2023
0.10
6.00
0.33
0.05
0.05
-0.09
00.5
+0.30
0.10
100
200
10
13 15
inverse-RIR
0.10
6.05
0.5
0.05
0.05
-0.0%
-0.8
-0.05
+0.30
156
-0.30
00.10
100
200
12 15
1.
26
forward-RIR
0.10
0.00
0.05
6.00
0.82
400
5.00
0.00
0.05
0.10
10
1.0
120
200
90
166
160
200
250
50
1G3
150
200
200
200
11
=
3
12
15
(3 27
inverse-RIR
8.30
0.0
0.10
0.05
0.00
0.50
18
1.00
0.00
-0.00
-0.55
0.05
0.10
1.0
253
100
100
230
100
200
11
=
12 = 18
13 = 35
forward-RIR
Figure IV: Two SoundNeRirF learned r2r-RIRs on MP3D dataset [7], we show two 12r-RIRs (split by the green line), each comprises of forward-RIR and an inverse-RIR, We explicitly show the time-shift learned by the each direct-path, specular-reflection and later-reverberaatin.
0.10
465
3.5
AAT-
0.05
-0.05
-0.10
100
200
inverse-RIR
0.10
315-
0.05
-800
00.05
-0.10
100
200
forward-RIR
Figure V: SoundNeRirF learned corridor-to-coridoor RIR. The left three sub-figures are learned specular, early and late reverberation RIR respectively.
corresponding result synthetic dataset shows the necessity and importance of each compoment of SoundNeRirF design.
In addition the three ablation studies, further conduct three more studies to figure out the necessity of setting different monotonicity constraints to inverse- RIR and forward-RIR respectively. Specifically, the necessity of increasing monotonicity inverse-RIR and decreasing monotonicity
18

---

Under review as a conference paper at ICLR 2023
0.05
0.00
0.05
-.43
-0.10
100
200
inverse-RIR
465
305
845-
0.05
400
-0.05
4.33
3.19
-0.10
100
200
forward-RIR
Figure VI: SoundNeRirF learned lab-to-corridor RIR. The left three sub-figures learned specular, early and late reverberation RIR respectively.
047
0.05
309-
000-
0.00
4/
0.33
180
30
3.39
750
300
-0.10
200
inverse- RIR
000
300
301
0.05
-0.05
-0
-0.10
100
200
forward-RIR
Figure VII: SoundNeRirF learned lab-to-lab RIR. The left three sub-figures learned specular, early and late reverberation RIR respectively.
to forward-RIR, We test two SoundNeRirF variants: both inverse- -RR and forward-Rie -RR are with increasing monotonicity constraints (SoundNeRirF jnnmmono) with decreasing monotonicity con- straints (SoundNeRirF (enaNeeRiiF_deemooo)) and swap the original constraint (SoundNeRirF swapmono, use decreasing monotonieity for inverse-RIR and increasing monotonicity for forward-RIIe RIR). The result is given Table III. We can conclude by this table that applying the same monotonicity constraints (ei- ther increasing or decreasing) both inverse-RIR and forward-RIR leads to inferior performance. Swapping the monotonicity generates the worst performance. It in turn shows the necessity of setting increasing monotonicity inverse-RRL and decreasing monotonicity forward-RIR.
C.4 MORE EXPERIMENT IN REAL-WORLD DATASET
In the main paper, we have reported the quantitative result on both one sound source and two sound sources setting. The result unheard sound test is given in Table IV. From this table, that SoundNeRirF performs the best for unheard sound prediction in real-world setting.
C.5 NOISE DISCUSSION
In real-scenario, robot position localization often suffers from noise, in which the robot cannot localize itself with high-precision. In our real-world sound data collection platform, Vicon localization system [32] gives very high-precision position coordinates of the robot at each position. To test the
19

---

Under review as a conference paper at ICLR 2023
Table III: Ablation Study on Monotonicity test
Methods
T-MSE (4)
TF-MSE (4)
CDPM (L)
SoundNeRirF incmono
0.30++0.00
0.05++0.01
0.47+00.00
SoundNeRirF decmono
0.31110.00
0.05110.01
0.48
SoundNeRirF swapmono
0.37:10.00
0.07:10.01
0.52.10.11
SoundNeRirF
0.19+00.00
0.03+00.01
0.43+00.1
Table IV: Unheard Sound Test on Real-World Dataset
LinInterp
0.37
0.07
0.93
NNeigh
0.62
0.10
1.02
Methods
T-MSE 10 (L))
TF-MSE (L)
CDPM (U)
WarpNet [48]
0.17+00.00
0.06:+0001
0.93=0.01
WaveNet [38]
0.03:10.01
0.01110.01
0.8910.01
TCN [2]
0.06:10.01
0.02:10.01
0.82.10.01
SoundNeRirF
0.02+#0.00
0.01+10.01
0.79=00.01
robustness of SoundNeRirF towards position noise interference, we specifically add noise pollution to Vicon provided position coordinates and test all models' performance position noise interference.
Specifically, we model noise as Gaussian noise with mean 0 and standard deviation a, 9 = 0, o)). Each coordinate x, y or is added noise sampled from the Gaussian noise. By varying the standard deviation a, we test all models' tolerance w.r.t. different level of noise interference. In this experiment, choose three noise levels, with 0.5, 1.0, 1.5, respectively. The result is shown in Table V, from which we can see that SoundNeRirF exhibits stronger rubustness towards noise interference than three comparing methods.
D MORE DISCUSSION ON EXPERIMENTAL SETTING
For each receiver position [z:, y. =], we first normalize each coordinate into [-1, before feeding to position encoding module, as WarpNet [48] does. In position encoding, the corresponding position encoding embedding size is 384. The learned time shift parameters for both the inverse-RIR and forward-RIR are quite small (both are less than 5).
Unheard Sound Test Explanation. In our experimental settings, in each sound source and receiver positions, collect two sounds. While one sound is used for training the model, the other sound is hidden for training but instead is used for testing. Since the two sounds theoretically share the same receiver-to-receive RIR, such "unheard sound test" helps test each model's generalization capability in predicting new sound.
Position Encoding since deep neural networks tend to learn lower frequency functions [43], follow the practice in 3D protein structure modelling [63], Transformer network [56] and RGB image based neural radiance field [33] encode the the low-dimensional position into high dimension. For example, for the reference position pr (also for the two virtually set positions 2 and 3 pr), the position encoding process works as,
PosEncode(pp.) [sin(2 mp,), cos((2 mp.), (sin(2--1Tpp)) kpr), cos -p,))
where p, = [zr, yr. ]. and L is the position encoding length, we use 64 in our work (so the position embedding size 384). The target position Pt goes the same process.
20

---

Under review as a conference paper at ICLR 2023
Table V: Noise Test on Real Data. In each evaluation metric entry, from left right the four values
correspond to metrics with noise standard deviation o = 0.0, 0.5, 1.0, 1.5 respectively. The standard deviation for each metric within 0.02.
Methods
T-MSE (10
+) (L)
TF-MSE (1)
CDPM (L)
WarpNet [48]
3.10
3.10
3.17
3.25
0.10
0.13
0.15
0.17
0.85
0.87
0.88
0.89
WaveNet [38]
2.05
2.07
2.09
2.12
0.06
0.07
0.09
0.11
1.17
1.18
1.19
1.19
TCN [2]
2.31
2.31
2.32
2.33
0.07
0.08
0.08
0.10
0.96
0.96
0.98
0.99
SoundNeRirF
1.20
1.20
1.21
1.22
0.05
0.05
0.06
0.71
0.72
0.72
0.74
Table VI: SoundNeRirF Network Architecture. The format FC@nI,n2 indicates a full-connection layer with input neurons nl and output neurons n2. Batch normalization (BN) is applied after each full-connection layer.
Reference Position MLP
Target Position MLP
Input: Ref. Pos. [B, 3]
Input: Target Pos. [B. 3]
Position Encoding
[B. 384]
[b 384]
TC@384, 512
TC4384,3 512
Layerl Communication
FC@512, 512
FC@512, 512
Layer2 Communication
FCW512, 512
FC0512,512
Layer3 Communication
FC@512, 512
FC@512, 512
Layer4 Communication
FC@512, 512
FC FC@512, 512 512 Layer5 Communication
FC@512, 512
FC@512, 12,512.55 512
time-shift head: FC@512,
time-shift head: FC@512,
RIR head: FC @ 5 257
RIR head FC 5522, 257
Output: inverse- RIR [B, 257]
Output: forward-RIk [B, 257]
E SOUNDNERIRF NETWORK ARCHITECTURE
SoundNeRirF network architecture is given in Table VI. In the layerwise communication, a trainable weight is used merge the intermediate features arising from reference position MLP (e.g. fi) and target position MLP (e.g. f2) before feeding them to the next layer, respectively,
[21 = w. [al
(8)
F EXPLANATION ON FIG. IN MAIN PAPER
Table VII: TF-MSE ()) variation change w.r.t. Reference-Targee Target Distance (corresponds to Fig. middle figure in Fig. main paper)
Methods
Close
Medium
Far
WarpNet [48]
0.35
0.41
0.42
WaveNet [38]
0.08
0.08
0.09
TCN [2]
0.15
0.14
0.14
SoundNeRirF
0.03
0.03
0.03
In Fig. 5 of the main paper, the curves in the middle and right figures are not associated with the exact value, and we re-scaled the y- axis for better comparison. To present more quantitative comparison, provide the exact value for each curve. The result of TF-MSE variation change w.r.t. reference- target distance is given in Table VII. The result TE-MSE variation change w.r.t. sound source number is given in Table VIII.
21

---

Under review as a conference paper at ICLR 2023
G PREDICTED SOUND SAMPLES
We also provide predicted sound samples by each method with our real-world dataset, as well as the ground truth sound in the supplementary material. The predicted sound samples use the reference sound in the lab area and target sound in the corridor area.
Table VIII: TF-MSE (!) variation change w.r.t. Sound Source Number (corresponds to Fig. 5 right figure in Fig. 5 main paper)
Sound Source Number
Method
3
4
5
WarpNet [48]
0.91
1.18
1.69
1.87
2.25
3.77
5.84
WaveNet [38]
0,02
0.05
0.09
0.09
0.11
0.12
0.12
TCN [2]
0.02
0.05
0.14
0.26
0.27
0.29
0.31
SoundNeRirF
0.01
0.01
0.02
0.03
0.04
0.05
0.07
H POTENTIAL SOUNDNERIRE APPLICATION IN ROBOTICS
The motivation of this work is to enable the robot to efficiently predict the sound to be heard at novel positions, without reply enough prior information about the room geometric features/sound source installation and rigorous and complex mathematical derivation. The robot can efficiently and effectively learn such neural acoustic field by simply exploring a new environment record the position and received sound at each step.
We further show that learning such sound neural field through our proposed SoundNeRirF framework a robot. Our experiment shows our method just needs relatively small
is easy to implement number of receiver recordings to learn an efficient sound neural field (Fig. left-most figure). So the robot doesn't 't have to walk a long time (exhaustive walking). At the same time, SoundNeRirF is liggt, achieves real-time inference even on resource-constrained edge device, For example, in the Table 6, SoundNeRirF takes 0.066s to predict the sound on Raspberry Pi. This advantage enables the robot to predict/render sound real-time, so fits for tasks that lay much emphasis on efficiency.
We argue that the learned SoundNeRirF can be used for various down-stream robot-related tasks. As far as be used to improve robot seff-localization capability, by comparing the predicted sound and truly recorded sound at particular position;" 2. SoundNeRirF can be used for embodied robotics research in synthesized audio-vision environment. For example, the audio-vision platform for embodied robotics SoundSpaces 1.0 just provides discrete and pre-computed RIR, so the whole environment is not dynamic and the robot can only hear sound at particular positions. SoundNeRirF can help to avoid such obstacles, enabling the robot render sound at arbitrary positions. VR/AR applications for immersive acoustic experience (audio auralisation).
at current moment, it includes but is not limited to: 1. SoundNeRirF can
22


Reviews:
[{"review_id": "qvFec-DkRr", "reviewer": "Reviewer_2NMU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In any given acoustic enclosure or a room, the sound produced at a certain location undergoes multiple reflections and travels a certain distance before arriving at a receiver kept elsewhere in the room. Because of this, the sound recived at a distant receiver is not the same as the sound at the source. One of the major problems in room acoustics is estimating the sound at a receiver at a known location given the original sound and the location of the sound sources under a specified room geometry. Typical signal processing methods employ ray tracing or wave tracing methods to account for reflections and under the linear time-invariant (LTI) model of a room, the impulse response between a source and a receiver with known locations (RIR) is estimated. The signal at the reciver would then be a simple convolution of the impulse response with the original source of sound. \n\nRecently, several authors have proposed using deep learning for estimating the RIR. The main drawback of such approaches is the need for a large labelled training set which maps source and receiver location to a specific RIR and this network would be trained specifically for a given enclosure. Different enclosures would need different networks. \n\nIn this work the authors propose to estimate the RIR between any two random locations based on sampling the room with up to 500 locations using a robot which would move to each location and record the audio along with receiver location. The authors propose to use this data to compute RIR between two receiver locations, instead of computing RIRs between a source and receiver using an MLP and a curious set of design choices. The paper is not very well written and needs effort to understand.  ", "summary_of_the_review": "Overall, Because of the Weakness pointed out in the above section I believe the design and problem formulation are flawed and the presentation is unnecessarily complicated. The evaluation is also not satisfactory. ", "strength_and_weaknesses": "Strengths:\n1. Interesting idea to estimate Receiver to receiver Impulse response. \n\nWeakness:\n1. The overall design simply does not make sense. Irrespective of the reverberation properties of the enclosure the authors propose to estimate RIRs of length 257 samples corresponding to 16 kHz sampling rate. This amounts to an RIR of duration less then 20ms. Typical residential rooms would have reverberation time (RT60) - measured as the time taken for sound to reduce to 60 db from its original value after the sound source has stopped. Thus longer RIRs are needed to even characterize such room attributes. The choice of using the same MLP to predict RIR of 2 other virtual distances which twice and thrice farther apart is also not well justified and is presented with a hand wavy argument - simulating ray tracing method (ISM). It is not at all clear how the authors are calling whatever the network is estimating as RIR as there is no evaluation comparing this to actual RIR. \n2. Essentially this approach is mapping a location in space to an RIR using an MLP. This is an multi-valued function and as such admits multiple RIRs for the same location based on the source location which is ignored in this work. \n3. It is assumed that the robot would move to each location and capture the exact same audio each time. It is assumed that is easy for the robot to record its location. This obviously means the source of sound is being precisely controlled to start and stop and required times to make sure the robot records when it is stationary. This problem setup does not make any practical sense. It is hard to imagine a case where  it is hard to also jot down the source location. With a known source location the estimation would be so much well defined. \n4. I also believe with the limited sound source chosen for evaluation (7 clips in simulation and probably only 2 in real data) the network is simply memorizing and generalization is not well studied at all. From the appendix it is clear that the training and test locations, although different are roughly in the same neighborhood. \n5. The overall presentation is rather confusing with terms like \"inverse-RIR\" and \"Forward RIR\" and it feels like these terms are set arbitrarily and is rather confusing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "kHGZCyhiZa2", "reviewer": "Reviewer_9Fm6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a deep model for learning to predict the sound that would be heard at a given target position, assuming that a reference position and the sound heard at the reference position are given. \n\nThe authors presents a MLP for learning forward RIR and inverse RIR, where the reference sound is first projected to a\nvirtual implicit sound source field with inverse-RIR, and then the virtual implicit sound source field is re-projected to the target sound with forward-RIR. \n\nThe authors also used three datasets to evaluate the performance of the proposed methods and benchmark them with several baseline methods, in terms of MSE, and TF-MSE. ", "summary_of_the_review": "Overall, this is a good paper with interesting ideas and some preliminary results. However, the paper also has several aspects for improvements which would lead the paper from good to excellent such as the validation of assumptions made. The visual results of RIR doe not seem to be promising although the MSE results look good.  ", "strength_and_weaknesses": "Strength of the paper: \nThis paper is overall well presented. The proposed idea looks ambitious but interesting. \nThe results in terms of MSE or TF-MSE look impressive. The authors also produced several datasets which they plan to release publicly and will be very useful for the research community.\n\nWeakness of the paper:\nThe authors consider the virtual microphone positions by simply multiplying a gain factor. The potential issues with this method include (1) this may not be a good way for dealing with early reflections and late reverberations; (2) how to choose these gain parameters in a practical scenario. In practice, the sound reflections from room surfaces can be from multiple directions. How can the gain parameters approximate these reflections or various time delays? Is there a principled way of choosing the gain parameters and are the two parameters a reasonable choice?\nThe authors showed the MSE and TF-MSE results, which can somewhat show how good is the performance of the compared methods, but may not be fully reflecting the truth. In the RIR shown in the paper, the early reflections seem to follow the direct sound immediately. I am expecting some time delays between the direct sound and early reflections, and also the early reflections are often more sparse than the late reflections. This does not seem to be case for the simulated RIR or the real room RIR. Not sure why the authors did not plot the ground truth RIR in the test case (considering the fact that they do have the ground truth RIRs that were used for computing the MSE). Showing both the estimated RIR and the ground truth can provide useful information on how good is the RIR learned by the network.     ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZfSi1SgoYl-", "reviewer": "Reviewer_tmuv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a supervised learning approach to learning the room impulse response function, specifically by leveraging multi-layer perceptron (MLP). Evaluation on a range of datasets shows that the proposed approach outperforms several recent approaches.", "summary_of_the_review": "The paper proposes a novel approach of learning room impulse response by leveraging a mobile robot to collect additional samples of the same source(s). Writing and clarity could be strengthened as well as addressing some of the aspects highlighted above to make it a strong paper ", "strength_and_weaknesses": "Strengths\n1. Evaluation across a variety of datasets.\n2. Proposed method significantly outperforms comparative baselines.\n3. Promised release of the referenced datasets.\n\n\nGrowth Opportunities\n1. I don't understand this statement. \"We also assume all receivers record the same sound content, so the sound at any target position can be inferred by the sound from any reference position.\" Isn't the sound received a function of the location and the propagation characteristics of the room?\n2. The motivation of using specifically 2 virtual receiver positions in 3.2 is unclear.\n3.  The drawbacks of classical RIR estimators is misleading as the proposed approach required significant number of labeled pairs for training, a key limitation.\n4. No justification is provided for the design of the specific MLP in Section 3.2 is provided. It would be also helpful to understand how sensitive the solution is to the architecture.\n5. Results of figure 5.a is confusing. If the proposed approach is truly independent of the number of receiver positions, it would be worth establishing how low the number could be. \n6. Additional testing on other room sizes and configurations would be helpful, especially with respect to different absorbent/reflection surfaces.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FcYB5_4GugX", "reviewer": "Reviewer_cCRH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposes a receiver-to-receiver sound neural room acoustics rendering field, which is different with previous works focusing on source-to-receiver RIR prediction. To be specific, the author decomposes the receiver-to-receiver RIR prediction into two parts: inverse-RIR and forward RIR, which is a very interesting design.", "summary_of_the_review": "The paper is novel and solid. However, there are still some missed details that can be further clarified.", "strength_and_weaknesses": "Strength:\n1. The paper solves the RIR prediction in a novel view: modeling the sound field with some positions with acoustic information, which is somewhat a interpolation method. Receiver-to-receiver sound field modeling requires no knowledge about the source and can support more complex scene.\n2. The author decomposes the receiver-to-receiver RIR prediction into two parts: inverse-RIR and forward RIR, which is a very interesting design and has physical intuition.\n3. The experiments are convincing and adequate, consisting of several datasets with different source number. The ablation verifies the effectiveness of each proposed technique.\n\nWeakness:\n1. Since the impulse response is the response of an impulse, it is a time-delay function (i.e., the impulse response assumes the impulse is emitted at the time 0, and the receiver gets the signal at positive (delay) timestamp). When we predict the waveform for a new position p_new, we convolve the waveform of a known position p_known with two IRs (inverse-RIR and forward RIR). In this case, the waveform will be further delayed, which means the method cannot handle the case that p_new is closer to the source than p_known. Maybe there are some tricks on applying the convolution?\n2. In the section 3.4 (second paragraph), it seems that some subscripts are wrong (e.g., t^inv_d < t^inv_d < t^inv_l), which makes the physical constraint hard to understand. Can you state it more clearly?\n3. Although the space limitation can be a reason, I think it is necessary for a paper to have a conclusion.\n4. Just a question, will the dataset used in the paper be released to make the future work on this direction easier?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject