Neural Synthesis of Binaural Speech From Mono Audio
Authors: Alexander Richard
Dejan Markovic
Israel D. Gebru
Steven Krenn
Gladstone Alexander Butler
Fernando Torre
Yaser Sheikh
Venue: ICLR 2021 Oral (2021)
Keywords: binaural audio
sound spatialization
neural sound synthesis
binaural speech
speech processing
speech generation

Abstract: We present a neural rendering approach for binaural sound synthesis that can produce realistic and spatially accurate binaural sound in realtime. The network takes, as input, a single-channel audio source and synthesizes, as output, two-channel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source. We investigate deficiencies of the l2-loss on raw waveforms in a theoretical analysis and introduce an improved loss that overcomes these limitations. In an empirical evaluation, we establish that our approach is the first to generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study. Dataset and code are available online.

Published as a conference paper at ICLR 2021
NEURAL SYNTHESIS OF BINAURAL SPEECH FROM MONO AUDIO
Alexander Richard, Dejan Markovic, Israel D. Gebru, Steven Krenn, Gladstone Butler,
Fernando de la Torre, Yaser Sheikh
Facehook Reality Labs Piusburgh, USA
(richardalex, dejanmarkovin www.rrooiiiiiiieiiioioo idgebru, stewennreeette gabut yaser
ABSTRACT
We present a neural rendering approach for binaural sound synthesis that can pro- duce realistic and spatially accurate binaural sound in realtime. The network takes, as input, a single-channel audio source and synthesizes, as output, two- channel binaural sound, conditioned on the relative position and orientation of the listener with respect to the source. We investigate deficiencies of the fz-loss on wave-forms in theoretical analysis and introduce improved loss that over- comes these limitations. In an empirical evaluation, we establish that our approach is the first generate spatially accurate waveform outputs (as measured by real recordings) and outperforms existing approaches by a considerable margin, both quantitatively and in a perceptual study. Dataset and code are available online
INTRODUCTION
The rise of artificial spaces, in augmented and virtual reality, necessitates efficient production of accurate spatialized audio. Spatial hearing (the capacity to interpret spatial clues from binaural signals), not only helps us to orient ourselves in 3D environments, it also establishes immersion in the space by providing the brain with congruous acoustic and visual input (Hendrix & Barfieid, 1996). Binaural audio (left and right ear) even guides us in multi-person conversations: consider a scemario where multiple persons are speaking in a video call, making it difficult to follow the conversation. In the same situation in a real environment we able effortlessly focus the speech from an individual (Hawley et al.) 2004). Indeed, auditory sensation has primacy over even visual sensation as an input modality for scene understanding: (1) reaction times are faster for auditory stimulus compared to visual stimulus (Tose & Praveen, 2010) (2) auditory sensing provides surround understanding of space as opposed the directionality of visual sensation. For these reasons, the generation of accurate binarual signal is integral to full immersion in artificial spaces.
Most approaches to binaural audio generation rely on traditional digital signal processing (DSP) techniques, where each component - head related transfer function, room acoustics, ambient noise is modeled as linear time-invariant system (LTT) (Savioja al.) 1999: Zotkin al. 22044 Sunder et al. 2015) Zhhng et al. 2017). These linear systems are well-understood, relatively easy to model mathematically, and have been shown to produce perceptually plausible results reasons why they are still widely used. Real acoustic propagation, however, has nonlinear wave effects that are not ap- propriately modeled by LTI systems. As a consequence, DSP approaches do not achieve perceptual authenticity in dynamic scenarios (Brinkmann et al. 2017), and fail produce metrically accurate results, i.e., the generated waveform does not resemble recorded binaural audio well.
In this paper, we present end-to-end neural synthesis approach that overcomes many of these limitations by efficiently synthesizing accurate and precise binaural audio. The end-to-end learning scheme naturally captures the linear and nonlinear effects of sound wave propagation and, being fully convolutional, is efficient to execute on commodity hardware. Our major contributions are (1) a novel binarualization model that outperforms existing state of the art, (2) an analysis of the shortcomings of the folsoss on raw waveforms and a novel loss mitigating these shortcomings, (3) real-world binaural dataset captured in non-anechoic room.

---

Published as a conference paper at ICLR 2021
Neural Time Warping
Temporal
PLTT
warp
(
Laver
HyparCoom
HyperConne
V1:T
(9)
21:T
warp activation
V1.T
meteral waep
geometric warp
C1:T
Figure 1: System Overview. Given the source and listener position and orientation C1:T at each time step, single-channel input signal 11:T is transformed into a binaural signal. The neural time warping module learns an accurate warp from the source position to the listeners left and right ear while respecting physical properties like monotonicity and causality. The Temporal ConvNet models nuanced effects like room reverberations or head- and ear-shape related modifications to the signal.
Related Work. State of the DSP techniques approach binaural sound spatialization as a stack of acoustic components, each of which is an LTT system. As accurate wave-based simulation of room impulse responses is computationally expensive and requires detailed geometry and material infor- mation, most real-time systems rely on simplified geometrical models (Välimäki al. 2012 Savioja & Svensson 2015). Head-related transfer functions are measured an anechoic chamber (Li & Peissig 2020) and high-qualit, spatialization requires binaural recordings at almost 10k different
spatial positions (Armstrong et al. (0118). To generate binaural audio the DSP-based binaural ren- derers typically perform series of convolutions with these component impulse responses (Savioja it al.] 1999 Zotkin et al. 2004 [Sndder et al. 2015 Zhang et al. 2017). For more detailed discus- sion, Appendix|A. [A.4
Given their success in speech synthesis (Wang et al., 2017), neural networks gained increased at- tention for audio generation recently. While most approaches focus on models in frequency do- main (Choi et al. 2018 Vasquez & Lewis, 2019), raw waveform models were long neglected due to the difficulty to model Long-range dependencies on high-frequeng audio signal. With the success of WaveNet (Van Den Oord et al. 2016) however, direct wave-to-wave modeling is of increasing interest (Fu et al. 2017] Luo & Mesgarani| 2018 Donahue et al.) 2019) and shows major improve- ments in speech enhancement (Defossez et al., (020) and denoising (Rethage et al., 2018). speech synthesis (Kalchbrenner et al. 2018), and music style translation (Mor et al. 2019).
More recently, first steps towards neural sound spatialization have been undertaken. Gebru et al. (2021) showed that HRTFs can be implicitly learned by neural networks trained on raw waveforms. Focusing on predicting spatial sound conditioned on visual information, a work by Morgado et al. (2018) aims to spatialize sound conditioned on 360 video. Yet, their work is limited to first order ambisonics and can not model detailed binaural effects. More closely related
line of papers originating from the 2.5D visual sound system by Gao & Grauman (2019)). In this work, binaural audio is generated conditioned on a video frame embedding such that object locations can contribute to where sound comes from. Yang et al. (2020); Lu et al. (2019): Zhou et al. (2020) build upon the same idea. Unfortunately, all these works have common that they pose the spatialization task as an upmixing problem, i.e., their models are trained with a mixture of left and right ear binaural record- ing as pseudo mono input. By design, these methods fail to model time delays and reverberation effects caused by the difference between source and listener position.
2 A NEURAL NETWORK FOR BINAURAL SYNTHESIS
We consider the problem where a monaural (single-channel) signal .1.7 = (II,..., IT) of length (l)
I' be transformed into binaural (stereophonic) signal vi.7, v17 representing the listener's left ear and right car, given conditioning temporal signal C1.7. This conditioning signal is the position
and orientation of source and listener, respectively. Here 3t, and correspondingly y/ (1) and y; are scalars representing an audio sample at time f. In other words, we aim to produce a function,
(r)
(v (1) yi
" f(II-Ax(((-A)))
where A a temporal receptive field. Each c, E R contains the 3D position of source and listener (three values each) and their orientations quaternions (four values each). Note that in practice,
2

---

Published as a conference paper at ICLR 2021
(t/r)
often is of lower frequency than the input and output signals .1. and Vir source and listener positions would likely not be updated at 48kHz but rather at typical camera frame rates such as 30- 120Hz. To simplify notation, we assume that has already been upsampled to the same temporal resolution as the audio signals.
Our overall framework is shown in Figure | A neural time warping module first warps the single- (2/r)) channel input signal 21:7 into a two-channel signal 11:7" where the channels represent left and right ear. The time warping compensates for coarse temporal effects and differences in time of sound arrival at left and right car caused by the distance between source and listener. The second block in Figure D is a stack of N layers, each of which is conditioned hyper-convolution (see Section 2.2 followed by a sine activation, which has been shown to be beneficial for modeling higher frequencies (Sitzmann et al.) 2020). Following the design of WaveNet, we use kemnel size 2 and double the dilation factor in each layer to increase the receptive field. This temporal ConvNet models nuanced effects caused by room reverberations, head and ear shape, or changing head orientations.
2.1 NEURAL TIME WARPING
Time warping is the task of mapping source temporal sequence onto target sequence and has long tradition in temporal signal processing, Most prominently, dynamic time warping (DTW) finds application in tasks like speech recognition (Juang. 1984) audio retrieval (Deng & Leung 2015).
DTW can be characterized as finding a warpfield Pı. that warps a source signal T15T to a target signal T1.T such that the distance between the signals is minimized.
PI. = arg min Pir
Σ
|
xp where Pt E (1,...,7), ,),
(1)
where the warpfield is typically constrained to respect physical properties such monotonicity (pt 2 Pt-1) and causality (pt S. t). For binaural audio, there is clear monotonous and causal relationship between source and target signal but the target signal is unknown at inference time. Additionally, the warping from mono to binaural signals goes far beyond simple linear time-shifts. For example, consider the source moving from the front to the left of the listener. This causes the delay between source and left ear to decrease but the delay between source and right ear to increase. source and/or listener are moving, other wave effects such as the Doppler effect influence how the signal needs to be warped from the source to the listener's left and right car. We are therefore interested in estimating a warpfield from the conditioning input Ci.r, i.c., from the spatial position and orientation of source and listener. A simple, parameter-fre approach is geometric warping based on the speed of sound Vsound and the distance between source and listener. Let Letp, I (which are part of Ct). Then, (gcom) = Pt
(Istn) be the source and listener positions at time (sre) and pt
( audio sample rate
lpp
(sr))
pl
(2)
Vounnd
This approach, however, fails to model important nuances such as the displacement between the left and right ear or diffraction delays as sound travels around the listener's head rather than straight through. In order to correct for those effects that geometric warping can not model properly, we estimate a neural warpfield PTT Figure
(ncural) WarpNet(cI.r) and add it to the geometric warpfield (cf.
= o (war)) (p) with P. = Pt
(neural) (geom)
where ((war)) (p:-1, P.) max(p:-1- min(t, pr)) is recursive activation function that ensures monotonicity and causality. The WarpNet shallow temporal convolutional network with four layers and 64 channels each.
The warped signal can now be computed using the predicted warpfield. Since the warpfield elements P: are typically not integers, we define the warped signal æ1. to be the linear interpolation of the original signal F1.7 at positions [pt] and [pt].
it ([pe] Pt) [( (pe (p:l) P[p=]-
(4)
In practice two warpfields are generated, one for each ear. Note how we explicitly enforce physical constraints in the warping by (warp)) min(t, (t) ensures causality by enforcing that the 1-th element
3

---

Published as a conference paper at ICLR 2021
of the warpfield not be larger than itself. Monotonicity is enforeed by max(pi-1:->> )): if an element has been warped from Pt-1 to position - 1, the next element at position must be warped from Pt-1 1 succeeding position. In contrast to related approaches such deformable convo- lutions (Dai et al., 2017) and spatial transformer networks (Jaderberg et al.) 2013). our neural time warping therefore allows for constrained warping of input signals with arbitrary lengths and directly models a physical phenomenon of sound.
2.2
CONDITIONED HYPER- CONVOLLTIOS
Raw waveform models where the output depends input signal and an additional conditioning temporal signal have primarily been studied in speech synthesis (Van Den Oord et al. 2016). The predominant approach towards such conditional temporal convolutions is additive combination of the input signal æ1. and the conditioning signal Ci.T, i.e., 21:7 that the result of the convolution at time is
W * * + C1.7 + b, such
K V CG-++++
Zt
W
(5)
Here, W E RCaxCuxX and V RConxCoull K are tensors containing the weights for temporal convolutions of the -indinnnsional input signal T1.T and the dimeeniinnii conditional signal CI.r with kernel size of K. Note that the convolutional weights W and V in this formulation are constant over time. Binaural filters in traditional digital signal processing, on the contrary, depend on the position of the sound source.
Inspired by the DSP formulation, we predict convolutional weights for the input æ layer and the bias as functions of the conditioning input CI.T.
K
21
( (Ct:)) (c.r)).... 0t- + (ct.))
(6)
This formulation is similar to the use of hyper-networks in Ha et al. (017)) but rather than generating them from intermediate feature maps, weights are generated from the conditioning input C1.r that contains physical information about the relation between source and listener. H(W) and H(6) are small convolutional hyper-networks that receive Cıa as input and predict the convolutional weights and the bias their output, respectively. Therefore, only is the input to the convolutional layer temporal sequence but the weights and biases change over time as well. We show in Appendix A.3 that if (W) and (6)) are linear networks, hyper-convolutions equal equation 5 plus a biliear term.
2.3 DEFICIENCIES OF THE 22-1OS ON Raw WAVEFORMS
Training a generative audio model with an fz-loss on the raw waveform is generally considered to result in poor audio quality and distorted signals particularly for speech. Therefore, a number of mostly spectrogram oriented alternative loss functions have been introduced over recent years (Kol-) bæk et al. 2020). Here, we provide an analytical explanation for a fundamental problem of phase estimation with the Ezlloss the waveform and show that simple additional loss term mitigates the problem. While correct phase estimation is not critical for single-channel audio, it is crucial for binaural audio
ears sensitive to interaural time differences small as 10ps (Brown & Duda 1998). To start the analysis, let
Ca (1:)) = Σ(ve
(ve D1
be the time-domain ly-loss between the predicted audio signal 91.1 and the target fur and let Yk. YE E C denote the k-th frequency component of yıut and fıer in the Fourier domain. We denote the amplitude error and angular phase error of the k--th frequency component as
L (amp) (Yz Yx)
||al| - pivl and (phasee (Yr, <(Yn, Px),
(8)
where the modulus (or magnitude) of the complex number.
4

---

Published as a conference paper at ICLR 2021
1:75
1.4
(target coerpo)
1.50
0.8
1.25
1.00
64-
0.75
0.4
1.75
2.00
0.0
1.75
(fa=losee)
(a) By (X,
(b) By (y (X,
Figure 2: Expected amplitude and phase error from Lemma I as a function of Êz-value E and target signal energy |
Lemma 1. Let Y E C be a fixed complex number and Y E complex number that has distance from Y. Then, the expected amplitude error and the expected angular phase error with respect to y are
be any
(Y, x)) Ey (C(au)) (
1
1 ddp
and
2T
Re
è e
+1
By (C(phase) (v,8)) ((.(pbas)((
1
arccos
dip.
(10)
2T
E/P
Proof. See Appendix A.I
Using Parseval's 's theorem, we write the time-domain f2-loss as the lossoss on the complex spectrum,
Σ Y 8g12.
(11)
Now, consider a single summand from equation 11 and denote the distance Y - Yy as e. Lemma allows analyze the expected amplitude and phase errors along this k-th frequency component. In Figure 2 we plot equation and equation 10|as as a function of the f2-value and the target energy [8). There are two key insights. First, the expected amplitude error is low even for large values that is, in the early stage of training as long as the target signal has high energy (top right part of Figure The phase, on the contrary, is barely optimized at all early in training when the fo-loss is large, even for high energy components, see Figure Second, over the course training, i.e., when the f2-loss decreases over time, the expected amplitude eтror among all target energies decreases. The expected phase error, on the other hand, improves primarily for high energy components and mid- and low energy components tend to have poor phase accuracy even for small f--values.
The above analysis shows that optimizing raw waveforms with a time-domain f--loss leads strong focus fitting the amplitudes but accurate phase reconstruction falls short. Since the models have limited capacity, the training data usually can only be modeled well but phase
up to an f2-loss Emin- this Emin
not sufficiently small, the signal's amplitude be significant. This can be critical since small amplitude errors lead to a slight change in speech coloration but phase errors introduce perceivable distortions. To overcome the deficiencies of the time-domain flos in phase optimization, we add an explicit phase term to the loss function,
will always
Clyer, ivr)) = L2(yee, (ir)) + X (phase) (
(12)
where STFT(y1:)) is the short-term Fourier transform of the audio signal V1.r
We discuss alternative formulation of this loss that operates fully in frequency domain Appendix A.2

---

Published as a conference paper at ICLR 2021
1.2
0.45
1.2
0.45
Angaler Phose ginioor
prmpp)
0.35
Angalur Phane Elror
0.35
Amp Etroe
Amplirode
0.8
0.25
0.5
0.22
0.016
0.014
0.012
0.01
0.016
0.014
0.012
0.01
fg-loes
Fyelosx
(x) training with time-domain Ey-loss only
(b) training with time-domain Ey-loss + phase loes
Figure Development of phase- and amplitude error as the Ez-loss decreases during training.
Table 1: Comparison of commonly used losses for audio modeling to proposed phase loss.
waveform
power spectrum
phase spectrum (angular error)
(22 error x10")
(f2 error)
power spectrum phase copy
1.276
0.048
1.563
multiscale STFT
2.279
0.043
1.996
Si-SDR
0.798
0.222
1.507
cross entropy on y-law encoding
0.161
0.039
1.199
t
0.141
0.037
0.886
phase loss (equation 12)
0.167
0.048
0.807
3 EVALUATION
Dataset. We recorded total of hours of paired mono and binaural data 48kHz from eight different speakers, four male and four female. The listener is a mannequin equipped with binaural microphones in its ears. Participants radius and have an unscripted conversation with it. We used an OptiTrack system to track position and orientation of source and listener throughout the captures. To the best of our knowledge, this is the only in-the-wild (i.c., not recorded in an anechoic chamber) binaural dataset of such size. We use validation sequence and the last two minutes from each participant as test data and train the models on the remaining data. See Appendix A.5 for a more detailed description. Network A Architecture. The WarpNet architecture ral convolutional network consists of three sequential blocks. Each block is a stack of ten hyper- convolution layers with 64 channels, kernel size 2, and the dilation size doubled after each layer. We train models for 100 epochs using Adam optimizer. Learning rates are decreased if be- tween two epochs the loss on the training set did not improve. At inference, our model can produce binaural audio in real-time.
asked walk around the mannequin
circle with 1.5m
as described Section 2.1
The tempo-
3.1 LOSS EVALUATION
In order to empirically validate our findings from Section 2.3 we train our proposed network with 3 how the phase error and amplitude error develop during training as the time-domain En-loss decreases. The model trained with fo-loss only (Figure Bl) shows the behaviour that the analysis in Section 2.3 suggests: the amplitude is optimized aggressively, particularly the beginning in training when the Eossoss is still high. The phase, on the contrary, does hardly improve at all in the beginning and shows only moderate improvements as the loss becomes smaller. When training with time-domain fossoss and phase loss (Figure Bb), this effect is being compensated for. The amplitude is optimized less aggressively and phase improves from the beginning of training on.
time-domain fz-loss only and with the loss proposed in equation 12 Figure shows
Various audio losses have been proposed over time, ranging from optimizing the power spectrum only and copying the input's phase (Zhao et al., 2018; Gao & Grauman 2019a) over multiscale
STFT loss for high frequency and high time resolution Yamamoto et al. (020 to optimization of the
scale-invarian signal to distortion ratio (si-SDR, Le Roux et al. (2019): Heitkaemper et al. (2020);
Luo & Mesgarani 2019)). With the introduction of WaveNet for speech synthesis (Van Den Oord

---

Published as a conference paper at ICLR 2021
Table 2: Ablation study. The components of the proposed binauralization network improve phase and amplitude and thereby the overall loss in time-domain.
raw waveform
power spectrum (62 error)
phase spectrum (angular error)
(f2 x00))
(a)
vanilla temporal CNN
0.254
0.061
0.934
(b)
warping
0.206
0.061
0.849
hyper-conv
0.183
0.051
0.847
(d)
sine activation
0.167
0.048
0.807
(anggaan plase (ols0-0r00 (1100)
L.wo) (amp-itude crror)
D aapplttude ezroz
0.093
pluse erear
1.4
recorded
0.080
1.2
0.067
1.0
0.053
0.8
grometzic
neural warping
neural
full
warping
warping +billnews
eystemm
(a) Warping example, Top to bottom: source mono input; left ear binaural recording; geometric warping in equation? 2 neural time warping in equation
(b) Amplitudc and phase error for different warping schemes, warping plus bilinear amplitude scale, and the full system. Figure 4: Analysis of the warping module.
et al. 2016), interpreting audio optimization as categorical optimization fıllaw encoded signal has become a prominent technique. As Table [ shows, all these approaches fail to predict accurate phase and mostly result in meager power spectral and waveform optimization. Overall, our proposed loss retains accurate & and power spectral estimations while outperforming other criteria by huge margin in phase error.
Perceptually, we observe strong correlation between the phase error and noise and distortions in the generated binaural signal. In particular, our proposed loss was the only one that produced clean speech without perceivable distortions. This is consistent with our perceptual study in Table where other approaches with different losses and architectures have been ranked less favorable.
3.2 MODEL EVALUATION
Ablation Study. In Table we show the impact of our model's individual components compared to vanilla temporal convolutional network baseline with WaveNet-like architecture and ReLU activations. Number of layers, channels, and kernel sizes are the same as in our final system. Keep- ing amplitudes unchanged but compensating for interaural time differences, is not surprising that neural time warping leads to a huge improvement in phase. Replacing regular convolutions with hyper-convolutions, on the contrary, particularly beneficial improve the power spectrum. Fi- nally, replacing the ReLU activations by sine functions, which have been proven to retain high fre- quency details more reliably (Sitzmann al. 2020), leads an additional moderate improvement along waveform, phase, and amplitude error.
Neural Time Warping. The purpose of neural time warping is strong initial alignment of the mono source signal to the left and right ear listener signal, respectively. Note the significant temporal shift between the mono signal and recorded left ear signal in Figure 4a the same figure, observe how geometric warping provides approximate alignment to the reference signal, while the learned neural warping successfully corrects the inaccurate geometric warping and aligns the peaks and valleys more accurately. Although those adjustments seem small, the impact of neural warping the phase error is significant, as shown in Figure 4b (red bars). Naturally, neural warping can not improve the amplitude (blue bars).
Temporal HyperConv Network. Neural warping provides an accurate alignment between input and target signal. This raises the question if deep network is required top of the warping

---

Published as a conference paper at ICLR 2021
0.011
standard convolution
L2 + AC (phaae)))
training loss
0.01
hyper convolution
0.009
0.008
0
20
40
60
80
100
epochs
Figure 5: Training loss of model with hyper-convolutions and model with standard convolutions. Hyper-convolutions lead to a significantly faster convergence.
Table 3: Comparison state of the art approaches for binaural sound synthesis.
waveform (&2 error x 10")
power spectrum
phase spectrum (angular error)
(E2 error)
DSP 2.5D Sound WaveNet
0.485
0.058
1.388
1.085
0.113
1.519
0.237
0.048
1.239
0.167
0.048
0.807
Table 4: Mean opinion scores of different approaches. Participants were ask to rank cleanliness, spatialization, and overall realism Likert scale from to 5.
cleanliness
spatialization
realism
DSP 2.5D Sound WaveNet
3.48
+ 0.88
3.75
+ 0.98
3.62
+ 0.90
2.70
1.09
3.18
0.94
2.70
1 1.03
1.20
0.51
2.92
1.11
ours
4.26
0.89
3.76
0.91
3.88
1 0.99
binaural recordings
3.69
0.94
3.88
0.96
3.82
0.88
module or if a linear amplitude adjustment can already yield convincing results. We therefore apply learned bilinear term the warped result,
(warped) zz
a aTc, b, a R. Coud IR
(13)
given the conditioning 6₂ and the warped signal (warped) for the left right ear, respectively. Fig- ure [bb shows that this Ieads to a slight improvement of the amplitude еттоr but falls way behind the performance of the full system with deep temporal network of hyper-convolutions after the warping module. Inspection of the mono and recorded signal in Figure 4a in fact reveals that the binaural recording undergoes additional transformations beyond warping. Room reverberations, source speech directivity, and modifications caused by the shape of the listener's 's ear, for instance, are physical effects that require complex transformations of the warped signal.
Many of these subtle effects depend on the position and orientation of source and listener in the room. It is therefore plausible that conditioned hyper-convolutions, which can model more complex dependencies between inputs and conditioning variables in single layer, show better performance than standard convolutions, ef. Table (b) versus (c). As Figure 5 reveals, hyper-convolutions also converge significantly faster than standard convolutions in the early stages of training.
3.3 STATE THE ART COMPARISON
In Table 3, we compare our approach to other neural binauralization approaches and to DSP base- line, which is the de-facto state of the art for binauralization. The recently proposed 2.5D visual sound (Gao & Grauman 2019b) network operates in frequency domain and predicts complex mask which the input is multiplied with to obtain left and right ear outputs. We compare to their approach and replace the visual features with conditioning features C1.7. For the STFT, use a window size of 1,600 samples and a hop length of 480 samples (10ms). Therefore, modeling delays of less than 10ms requires non-trivial manipulation of the phase information in the complex
8

---

Published as a conference paper at ICLR 2021
Table 5: Real-time-factor for offline processing and latency for streaming generation of binaural audio. The DSP baseline runs on CPU, all other models run on an NVidia Tesla V100.
trainable parameters
offline inference faatttmmefactor
streaming mode latency
DSP (* on CPU) 2.5D Sound
0.680
25.0ms
(1:2.1ms)
76.7M
0.013
WaveNet
1.9M
0.043
31.9ms
(+m).3ms)
ours
8.6M
0.069
32.8ms
(=:0.4ms)
spectrogram, which is more difficult operation than modeling delays in time-domain. We also provide a comparison to a WaveNet that proved to be generally strong in various generative audio problems (Rethage et al., 2018 Engel et al., 2017). For the conditioning sitions, we follow the approach of Van Den Oord al. (2016), i.e., the source and listener positions appended to the input of each temporal convolutional layer. Note that we use the WaveNet in non-autoregressive setup since the input audio, i.e., the mono signal, is fully available at inference time. Overall, our approach performs significantly better than other methods.
source and listener po-
For a perceptual evaluation, we asked 100 participants to rank a total of 2,000 audio snippets from to on Likert scale according to three criteria: cleanliness of the signal, spatialization quality, and overall realism, see Table 4 All scores are below a 5 (indistinguishable from reality) because par- ticipants listened results for a generic head-related transfer function rather to one that takes their explicit head and ear shape into account. Additionally, user's headphones are of different quality and not equalized. Note that the binaural (ground truth) recordings score lower on cleanliness be- cause they contain ambient noise that is uncorrelated to the source input and therefore not modeled by our approach. WaveNet leads particularly noisy audio signal, which is caused by two major factors. First, audio is generated at 48kHz, which is more difficult to model than 16-24kHz audio. Arik et al. (2017) show that the quality of WaveNet degrades with higher sampling rates. Second, WaveNet has to spend a considerable amount of capacity on modeling large source-to-listenr time shifts between the mono and binaural signals and, in consequence, struggles more to generate clean audio. Our approach ranks favorably against other neural binauralization systems and is also pre- ferred in terms of cleanliness and realism over the DSP baseline. A t-test showed that all results in Table statistically significant with the exception of spatialization between ours and DSP, which is ranked at almost equal quality. Since DSP is the perceptually closest competitor to our approach, performed an additional perceptual side-by-side study between the two systems that comfirms the results presented in Table 4 see Appendix A.6
Runtime, When analyzing the runtime of system, two cases are important. The first is offline processing, where a user provides the complete mono audio to be binauralized in advance. The real-time-factor is the computation time divided by the duration of the input. Table E shows that our system allows for rapid binauralization. On a single NVidia Tesla V100, our approach can binauralize 100 seconds of mono audio in just 6.9 seconds. Note that the DSP baseline does not run on GPU but is purely CPU-based. The second case is a streaming scenario where binaural audio has to be computed on-the-fly, e.g., when a user navigates through 3D environment in a game or in virtual reality. In this case, systems required to have low latency. On an NVidia Tesla V100, our approach runs with roughly 33ms latency, which is low enough to have non-observable delays for videos or games that render frames at 30Hz. Note that this is measured with a naive pytorch implementation and allows for several improvements to further lower the latency. The 2.5D sound network can not efficiently be applied in streaming mode because (a) it is acausal, i.e., requires access to future audio, and (b) it summarizes as much as 0.32 seconds of audio in a single temporal step in its bottleneck layer due to the UNet structure.
CONCLUSION
Our neural sound binauralization approach is the first purely data-driven end-to-end model that shows convincing performance compared to traditional state of the art binauralization methods. We were able to show effectiveness of our model both quantitatively and in a perceptual user study. Moreover, we unveiled and mitigated a fundamental issue with &g -p-otiimznttion on the raw waveform that affects not only this task but is relevant other generative audio problems as well.

---

Published as a conference paper at ICLR 2021
REFERENCES
Jont B. Allen and David A. Berkley, Image method for efficiently simulating small-room acoustics.
The Journal of the Acoustical Society of America, 65(4):943-950, 1979.
Serean O Arik, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng. Jonathan Raiman, et al. Deep voice: Real-time neural text-to-speech. Int. Conf. on Machine Learning, 2017.
Cal Armstrong, Lewis Thresh, Damian Murphy, and Gavin Kearney, perceptual evaluation of individual and non-individual hrtfs: A case study of the sadie ii database. Applied Sciences, (11):2029, 2018
Futoshi Asano, Yoiti Suzuki, and Toshio Sone. Role of spectral cues in median plane localization. The Journal of the Acoustical Society of America, 88(1):159-1668 1990.
Durand R. Begault, Alexandra S. Lee. Elizabeth M. Wenzel, and Mark R. Anderson. Direct com- parison of the impact of head tracking, reverberation, and individualized head-related transfer functions on the spatial perception of virtual speech source. Journal of the Audio Engineering Society, 2000.
Samuel Bellows and Timothy Leishman. High-resolution analysis of the directivity factor and di- rectivity index functions of human speech. Journal of the Audio Engineering Society, 2019.
Piotr Bilinski, Jens Ahrens, Mark R. P. Thomas, Ivan J. Tashev, and John C. Platt. Hrtf magnitude synthesis via sparse representation of anthropometric features. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp.
2014.
Fabian Brinkmann, Alexander Lindau, and Stefan Weinzierl. On the authenticity of individual dy- namic binaural synthesis. The Journal of the Acoustical Society of America, 2017.
C Phillip Brown and Richard O Duda. A structural model for binaural sound synthesis. IEEE transactions on speech and audio processing, 6(5):476-488 1998.
Corey I. Cheng and Gregory H. Wakefield. Introduction head-related transfer functions (hrtfs): Representations of hrtfs in time, frequency, and space. Journal of the Audio Engineering Society, 49(4):231-249, 2001.
Hyeong-Seok Choi, Jang-Hyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Ha, and Kyogu Lee. Phase-aware speech enhancement with deep complex u-net. In Int. Conf. Learning Represen- tations, 2018.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Int. Conf. Computer Vision, pp. 764-773, 2017.
Alexandre Defossez, Gabriel Synnaeve, and Yossi Adi. Real time speech enhancement in the wave- form domain. In Interspeech, 2020.
James Deng and Clement HC Leung. Dynamic time warping for music retrieval using time series modeling of musical emotions. IEEE Transactions Affective Computing, 6(2):137-151, 2015.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Int. Conf. on Learning Representations, 2019.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Mohammad Norouzi, Douglas Eck, and Karen Simonyan. Neural audio synthesis of musical notes with wavenet autoencoders. In Int. Conf. on Machine Learning, pp. 1068-10771 2017.
Szu-War Wei Fu, Yu Tsao, Xugang Lu, and Hisashi Kawai. Raw waveform basedd speech enhancement by fully convolutional networks. In Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp. 6-12, 2017.
Ruohan Gao and Kristen Grauman. Co-separating sounds of visual objects. In Int. Conf. on Com- puler Vision, pp. 3879-3888, 2019a.
10

---

Published as a conference paper at ICLR 2021
Ruohan Gao and Kristen Grauman. 2.5d visual sound. In IEEE Conf. Computer Vision and Pattern Recognition, 2019b.
Israel D Gebru, Dejan Markovic, Alexander Richard, Steven Krenn, Gladstone Butler, Fernando de la Torre, and Yaser Sheikh. Implicit hrtf modeling using temporal convolutional networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, 2021.
Corentin Guezenoc and Renaud Seguier. Hrtf individualization: A survey. Journal of the Audio Engineering Society, october 2018.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In Int. Conf. on Learning Representattons, 2017.
Monica L Hawley, Ruth Y Litovsky, and John F Culling. The benefit binaural hearing in a cocktail party: Effect of location and type of interferer. The Journal of the Acoustical Society of America, 115(2):833- 843, 2004.
Jens Heitkaemper, Darius Jakobeit, Christoph Boeddeker, Lukas Drude, and Reinhold Haeb- Umbach. Demystifying tasnet: A dissecting approach. In IEEE Iut. Conf. Acoustics, Speech and Signal Processing, pp. 6359-6363, 2020.
Claudia Hendrix and Woodrow Barfield. The sense of presence within auditory virtual environments. Presence: Teleoperators and Virtual Environments, 5(3):290-301, 1996.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In Ad- vances Neural Information Processing Systems, pp. 2017-2025, 2015.
Shelton Jose and Kumar Gideon Praveen. Comparison between auditory and visual simple reaction times. Neuroscience & Medicine, 2010, 2010.
B-H Juang. On the hidden markov model and dynamic time warping for speech recognition unified view. AT&T Bell Laboratories Technical Journal, 63(7):1213-1243 1984.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. arXiv preprint arXiv: 2018.
Brian G. Katz. Boundary element method calculation of individual head-related transfer function. i. rigid model calculation. The Journal of the Acoustical Society of America, 110(5):24402444 24488 2001.
Paulina Kocon and Brian B. Monson. Horizontal directivity patterns differ between vowels extracted from running speech. The Journal of the Acoustical Society America, 144(1), 2018
Morten Kolbæk, Zheng- Hua Tan, Søren Holdt Jensen, and Jesper Jensen. On loss functions for pervised monaural time-domee domain speech enhancement. JEEE/ACM Transactions on Audio, Speech, and Language Processing,
2020.
Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. Sdr-half-baked well done? In IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 626-630 2019.
Song and Jürgen Peissig. Measurement of head-related transfer functions: A review. Applied Sciences, 10(14):5014, 2020.
Yu-Ding Lu, Hsin- Ying Lee, Hung- Yu Tseng, and Ming-Hsuan Yang. Sclf-supervised audio spatial- ization with correspondence classifier. In IEEE Int. Conf. on Image Processing, pp. 3347-3351, 2019.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 696-700, 2018.
Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time--freeeeee frequency magnitude masking for speech separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(8): 1256-1266, 2019.
11

---

Published as a conference paper at ICLR 2021
Noam Mor, Lior Wolf, Adam Polyak, and Yaniv Taigman. A universal music translation network. In Int. Conf. on Learning Representations, 2019.
Pedro Morgado, Nuno Nvasconcelos, Timothy Langlois, and Oliver Wang. Self-supervised gener- ation of spatial audio for 360 video. In Advances in Neural Information Processing Systems, pp. 362-372, 2018.
Sebastian Prepelitä, Michele Geronazzo, Federico Avanzini, and Lauri Savioja. Influence of vox- elization finite difference time domain simulations of head-related transfer functions. The Journal of the Acoustical Society of America, 13965):249--2504 2016.
Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In IEEE Int. Conf. on Acoustics, Speech and Signal Processing, pp. 5069-5073, 2018.
Atul Rungta, Carl Schissler, Nicholas Rewkowski, Ravish Mehra, and Dinesh Manocha. Diffrac- tion kernels for interactive sound propagation in dynamic environments. JEEE Transactions on Visualization and Computer Graphics, 24(4):1613-1622, 2018.
Lauri Savioja and U. Peter Svensson. Overview of geometrical room acoustic modeling techniques. The Journal of the Acoustical Society of America, 138(2):708-730, 2015.
Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Sociery, 47(9):675-705, 1999.
Barbara G. Bhinn-Cunningham, Norbert Kopco, and Tara J. Martin. Localizing nearby sound sources in a classroom: Binaural room impulse responses. The Journal of the Acoustical So- ciety of America, 11765):100031155, 2005.
Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. arXiv preprint arXiv:2006.0966 096611 2020.
Kaushik Sunder, Jianjun He, Ec-Leng Tan, and Woon-Seng Gan. Natural sound rendering for head- phones: Integration of signal processing techniques. IEEE Signal Processing Magazine, 32(2): 103, 2015.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Process- ing Systems, 33, 2020.
Vesa Välimäki, Julian D. Parker, Lauri Savioja, Julius O. Smith, and Jonathan S. Abel. Fifty years of artificial reverberation. IEEE Transactions on Audio, Speech, and Language Processing, 20 45):1421-4448, 2012.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. Wavenet: A generative model for audio. In ISCA Speech Synthesis Workshop, pp. 125, 2016.
Sean Vasquez and Mike Lewis. Melnet: generative model for audio the frequency domain. arXiv preprint arXiv:1906.01000 2019.
Yuxuan Wang, RJ Skenry-Ryan, Daisy Stanton, Yonghui Wu, Ron Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. In Interspeech, 2017.
Frederic L. Wightman and Doris J. Kistler. The dominant role of low www-freuueccy interaural time differences in sound localization. The Journal of the Acoustical Society of America, 1648- 1661, 1992.
Kazuhiko Yamamoto and Takeo Igarashi. Fully perceptual- 3d spatial sound individualization with adaptive variational autoencoder. ACM Transaction on Graphics, 36(6), 2017.
12

---

Published as a conference paper at ICLR 2021
Ryuichi Yamamoto, Eunwoo Song. and Jae-Min Kim. Parallel wavegan: A fast waveform genera- tion model based on generative adversarial networks with multi-resolution spectrogram. In IEEE Int. Conf. Acoustics, Speech and Signal Processing, pp. 6199-6203 2020.
Karren Yang. Bryan Russell, and Justin Salamon. Telling left from right: Learning spatial corre- spondence of sight and sound. In IEEE Conf. on Computer Vision and Pattern Recognition, pp. 9932-9941, 2020.
Wen Zhang, Prasanga Samarasinghe, Hanchi Chen, and Thushara Abhayapala. Surround by sound: A review of spatial audio recording and reproduction. Applied Sciences, 7(5):532, 2017.
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In European Conf. on Computer Vision, pp. 570-586, 2018.
Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In European Conf. on Computer Vision, 2020.
Dmitry N. Zotkin, Ramani Duraiswami, and Larry S. Davis. Rendering localized spatial audio in virtual auditory space. IEEE Trunsactions on Multimedia, 6(4):553-564, 2004.
13

---

Published as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF OF LEMMA I
Before we start with the formal proof, let us get a more intuitive idea what Lemma means and how phase and amplitude error relate to the Ey-loss in time-domain and complex spectral domain, respectively. Reconsider the fz-loss in time-domain and use Parseval's 's theorem to relate it to the complex frequency domain,
E(v+ - 0z)² (1)) Σ Vi - VE1,
(14)
where k runs over all frequency components of the signal. The time-domain E--loss is therefore sum of the f2-loss of each individual frequency component in spectral domain. For the analysis, let us consider one fixed frequency component Our findings hold for frequency components equally. Figure 6 illustrates this case in the complex plane P Given a target Ý and a prediction Y that has distance 5 to Y, optimizing the E2-loss equal to optimization of the amplitude (the length difference of Y and Y) By = 8y. If the phases fy and fg are different, however, the relation is that obvious. Lemma [ makes a statement about the expected amplitude and phase error for random prediction Y that has di-dintance to Y, i.e., that lies on the circle defined by B gy
Im
B
(Y7
197
Py.
Re
Figure 6: Graphical illustration of the premises for Lemma on the complex plane. Y the target, is prediction with distance e to Ý. The amplitude error is defined as | error is the difference between oy and 0y.
[8]| and the phase
With this in mind, prove Lemma
Let V e C a fixed complex number and Y -B any complex number that has distance frrmm Y. Then, the expected amplitude error and the expected
We drop the index and just write Y and V.
14

---

Published as a conference paper at ICLR 2021
angular phase error with respect to Y are
Ey Ec (C(amp) (Y,8)) = =I
Jyeb.
C (amp)) (V, (V,Y)
+ eA 11 -1
and
(15)
2n
By (C(phase) (Y, (r, = JYER,
C(phase) (Y, ())(())
c 4p +
1
arccOs
dip.
(16)
2m
chu
Proof. Let E C and Y E y. Then, amplitude loss and phase loss as defined in equation 8 are given by
((amp) (Y, | 1811.
(17)
(Y, = <(,,)) = arccos Re(Y) Re(Y) + Im Im(Y)Im(( (Y)
L(phase) (
(18)
|Y| Y
where denotes the modulus (or magnitude) of complex number.
Without loss of generality, let Im( 0 and Re(Y) 2 O. This can always be achieved by a unitary rotation of angle By around the origin of the complex plane which preserves the lengths and angles between Y and Y. In that case, the phase loss simplifies
Re(Y)- 11[
Re(Y) Y
C(phase) (Y,8)
arccos
arccos
(19)
Y |
Since Y E B B, is a point on the circle around Ý with radius & (see Figure 6 for an illustration), B y.y can equivalently be written as
= (YE: = + +8. --,
(20)
Then, each E Jy, uniquely defined by some &o [- n] and we can rewrite ((amp) as function of up by substituting Eeti Y,
(amp) (() = lEeff + 81 1111
(21)
E.
Observe that due to lm( and Re(r)= 2 0, follows that Re(Ý)= = | and
Iee 8 (E
+ + 1 + sin(
=181/
E²
+2
E
cos + 1
1112
e2
=181/
+
cos + cos(g)? + sin()))
1
Y
= -1
E |
+
(22)
Therefore,
(aamp) (4) = 18
+
t00 -
(23)
15

---

Published as a conference paper at ICLR 2021
For the phase error, we apply the same substitution and obtain
Re(eet + 8)
(phaasc) c,
(4))
= arccos
|EE'R + I
M
_ COS 4p + 1)
arccos
|
c + tetp
M
| Cos p +1
=
arccos
+ eXP
Re
etp + 1)
arccos
(24)
+ exP
|&|
The expected amplitude and phase error can now be written
Exp (Clamp)
(1))
5 clamp)
(p)p(1))
and
(25)
E,
Exp (ip 2.8
(w))
S c(phase) E,8
(p)p(1))dp.
(26)
Assuming that Y B.yy is uniformly distributed the circle around Y, i.e., p(p) follows a circular uniform distribution, we have p(y)) p(p)) = from Lemma
1 and plug in equation 23 and equation 24 to obtain the claim
2m
A.2 FREQUENCY-DOMAT LOSS FORMULATION
In our original formulation of the phase-enhanced loss in equation 12 phase is technically penalized twice: once implicitly in the fo-loss on the raw waveform and once explicitly in the phase loss term. It is possible formulate the phase-enhanced loss in frequency domain such that there is clear separation of magnitude and phase loss in two different additive terms.
Due to Parseval's theorem, optimizing the Enlloss in time domain is equivalent to optimizing the f2-loss in frequency domain. We transform the time domain audio signal into frequency domain using short-term Fourier transform and denote the result of the transformation as
YI.K,:: STFT(
(27)
where K the number of frequency bins of the discrete Fourier transform and S is the number of STFT steps, i.e., S T //hhp_Jength Since the frequency spectrum can be expressed by it's magnitude and phase spectrum, we can reformulate equation] 12 as
= Σ (Yo.p)(Yx... K.s
Ý2,s) + A (Yk, Yn.e)..
(28)
Note that this form is not equivalent to the original loss since separating the frequency spectrum into magnitude and phase is a non-unitary operation. However, we do not find the differences to be significant, as Toble In fact, the clear separation allows for better adjustment of the phase weight À and can even lead to slightly improved results.
A.3 INTERPRETATION OF HYPER-COM CONPER-COVOLLUIONNS
Hyper-convolutions are a generalization of stardard convolutions as long as H( and H(b) are able to learn a linear transformation. Specifically, if H(W) and (b)) are linear convolutional networks, hyper-convolutiooss extend equation 5 by bilinear term. We show this property in the following and provide an example for a simple fully connected layer.
Recall the definition of hyper-convolutiion from equation 6
K
21
(W
xt 0+1 + (crn).....
(CL.).
(29)
16

---

Published as a conference paper at ICLR 2021
Table 6: Comparison of the loss formulation from equation 12 and equation 28 While the first
penalizes phase twice, once implicitly in the time-domain fo-loss and once in the explicit phase loss term, the latter provides clear separation between magnitude and phase loss terms.
waveform (&z error x 10
power spectrum
phase spectrum (angular error)
(E2 error)
from equation 12
0.167
0.048 0.036
0.807
loss from equation 28
0.157
0.809
Lemma 2. Let H and HD be linear convolutional networks with kernel size K. Let further BuT and CUT be a sequence of input and conditional vectors in RCn and RC and the output Z1:T be sequence of vectors in RCs,, Then, the hyper-convolution, from equation 29 reduces to
K
K
Ca
It I+-
==
V Mc-++++
(30)
with U E C W C VE B Comxk and E RCw,
Proof. We start from equation 29 and use that both H(( and H() are linear (and therefore, w.l.o.g.
single-layer) convolutional networks with kernel size K, such that
21 (c.
x k+1 + (c-Kk2z
(31)
First, consider (())) I Cons X w Can, Since H()) is linear, it can be written
K
(c
Σ +
(32)
with V E RCan and RC Note that this already yields the last two terms in equation 30
While H(b) only generates bias, H(W) needs generate IaCaxCCnX sized tensor to realize the convolution with æI.. Thus, H(W) RCauxK R CCCxCax and due H(W) being linear function, the component at (i,j. k) of the output tensor is given as
W (c-
U + Wij.k.
(33)
where U E XCax K)x (CemaXX) K) is the weight tensor of the hypernetwork (W(W) and W e B.Cu x Or is its bias. For simplicity of notation, let & be 2, from equation 31 without H()) (C,- (c.K) Using equation 33 2, is then given by
2,
H (c. (c- Iz-A+1
Cn
K
U.jkn + W
I1-A+1.j
>=1j=1 #=
K
Cn
K W W.
Σ
2 - U.J.
(34)
=
111=
17

---

Published as a conference paper at ICLR 2021
Together with equation 32 this yields the claim from Lemma
2: Et + H1))
(c
(C-K++:s))
K
K
Ca
It-k+I
k=
#=1j=
V +.
(35)
The last row of equation 33 is exactly the definition of a standard conditioned temporal convolution from equation 5 The row before is a bilinear combination of C1:7 and æ1:7. As consequence of Lemma 2 conditioned (linear) hyper-convolution are therefore a strict generalization of the stan- dard conditioned temporal convolutions from equation Note that any non-linear HWW) and H(6) that is capable of learning linear transformation therefore also a strict generalization of equa-
tion 5
Example: K Cout
We illustrate the case for K = 1, i.e., the convolutions break down to a simple fully connected layer. To simplify notation, we also restrict this example to single output channel Cout
With linear hyper-convoltions and Cout = 1, we have
H( (c,) Uc, w, (c)) - c++b
(36)
for a weight matrix U E CCCCCoon and bias vector w E RC, as parameters of H(W) as well as weights v E &Coss and bias b E IR as parameters for H(b) Inserting this into equation 29) leads to
2 H( + (c)
v² c + =xTUc.+ + w x + T c +.
(Uc, w)
2, 220
(37)
Compared to the standard formulation from equation 5
w0 xt + ee C +b,
(38)
equation 37 adds the bilinear term 2! UC, while all other terms are still remaining.
A.4 SIGNAL PROCESSING BASELINE
arrives the left and right ear offset times due to the marginal difference in their distance from the source of sound (Wightman & Kistler 1992). The coronal (back/front) asymmetry of the outer ear (pinnae) further transforms the incoming sound wave differ- ently depending on the direction of the source (Asano et al. 1990 Cheng & Wakefield (001). Room effects such reverberation influence auditory localization as well Shiin--uuninnhham et al. 2005).
Sound produced in
These binaural disparities allow listeners to localize sources of sound in three dimensions (Begauli ct al. 2000) and gain a more complete sense of the state of the space around them.
Traditionally different effects that influence propagation between the source and the listener, i.e., different components the mapping function between the input mono signal and output binaural signals, addressed separatcly. The components are assumed to be linear time-invariant (LTI) systems and therefore completely characterized by their impulse responses. They are then combined to produce the output signals with series of convolution operations (Savioja al. 1999] Zotkin
et al. 2004 Sunder et al. 2013 Zhang et al. 2017). A short overview follows.
Source. People not omnidirectional sound sources and, unlike loudspeaker whose spatial di-
rectivity depends only on the frequency. the human directivity pattern may depend on speech content and pose well. Though some studies of speech directivity in controlled settings exist (Kocon &
18

---

Published as a conference paper at ICLR 2021
Monson 2018;
Bellows & Leishman 2019), in practical applications it is either ignored approx- imated with a simple pattern such as a cardioid.
Environment: room acoustics. The reverberation is caused by the interaction of the sound field with the surrounding environment. Different approaches to room acoustic modeling cxist but they mainly divided in two groups (Välimäki et al. 2012): (1) physically accurate but computationally expensive wave-based methods that, given detailed geometry and material information, numerically solve the wave equation; and (2) methods based on geometrical acoustics (Savioja & Svensson 2013) that ignore the wave nature of sound and assume a ray-like behaviour, and are therefore more suitable for real-time operation (wave phenomena such as diffraction is usually modeled sep- arately (Rungta et al.] 2018)). For the real-world rooms the room impulse responses (RIRs) are either measured or computed using simplified geometric models informed by some estimated room parameters. It is important to note that measurement procedure is infeasible for fully dynamic sce- narios since RIRs depend on both source and listener spatial positions. The length of the RIR filters depends on the reverberation time of the environment, i.e., the time it takes for sound to decay by 60dB, which can go from less than half a second for typical office spaces to a couple of seconds for auditoriums and concert halls, and around ten seconds for large cathedrals.
background noise. Even in absence of other interfering sound sources, there is always some degree of ambient noise present in the environment. Usually this noise is assumed to be diffuse and independent from the listener position. It is also often assumed to be stationary and it can be estimated from short silence intervals.
Environment:
Listener. The human body, most notably pinna, head and torso, modify the incoming acoustic waves in a way that is crucial for the spatial perception of sound. Traditionally the head-related transfer function (HRTF) is used to model these effects (Cheng & Wakefield (001). In theory, the HRTF is personalized
the individual (listener) and depends on the source position relative to the listener. However, most practical implementations use a generic (oot-personalized) HRTF though HRTF penalization is an active research area (Bilinski et al., 2014; Yamamoto & Igarashi 2017 Guezenoc & Seguier 2018). Moreover, measuring HRTF in a volume is impractical and it is usually measured on fixed radius (Li & Peissig, 2020). making the HRTF a function of the source direction only. In alternative, the boundary element method (BEM) (Katz) 2001) or finite-difference time-domain (FDTD) method (Prepelită et al. 2016) can be used for numerical HRTF simulation using head and torso scans. Depending on the dataset, the HRTF filters are usually 2.5 after removing the initial onset delays.
20ms long
Equipment.
While above components enough to model the physics involved in the mapping between the input and the output signals, in a practical setting the captre/reproducctinn equipment plays a role as well. To compensate for the frequency response of the equipment and the signal- processing chain, the equalization filter should be applied. It is often assumed that this filter does not change in time and can be estimated from a test capture.
Note that each of above steps introduces some degree of estimation, measurement or modeling errors that accumulate down the pipeline and, not being formulated in end-to-end fashion, the solution is sub-optimal from the perspective of the particular application. Furthermore, study showed that even using binaural room impulse responses measured for the test subject inside the test environment, the perceptual authenticity between virtual and real sound sources was not fully achieved for a dynamic scenario that allowed natural head movements of the listeners (Brinkmann) lt al. 2017).
Our implementation, used as the DSP baseline, computes the output binaural signals y (1/r) (4), for left and right ear respectively, from the input mono signal s0)), / being the sample index, as follows:
y (() heg (t))
hut (// (t, (/sc.btt)) (,, 0 Pr. (se) ((hnn)) xt) + w (t))
(39)
where
z(t) is assumed to be a clean input signal; * indicates the convolution operation;
hir(t, (fsre)
p (src)
Pt (smm) is the RIR computed using the image source method Allen &
Berkley 1979) assuming a simple rectangular room and reverberation time 0.2s; P. 186) and
19

---

Published as a conference paper at ICLR 2021
p( are the source and listener positions, and (, indicates the source orientation used to simulate cardioid directivity;
hon (t. @fatc.kam) is the head related impulse response (HRTF in the time-domain) for the left / right ear; the HRTF of a KEMAR mannequin, measured in an anechoic chamber at 9600 unique discrete positions on a sphere of radius 2m is used; of indicates the direction on which the source is found with respect to the listener's front:
beg heffr)() (t) is the equalization filter for the given channel, estimated from a test capture in presence of the source signal; and
(1) is the random noise added to the given channel, generated with a power spectral density estimated from a test capture during a silence period.
Since the filters depend on source and listener positions, which in dynamic scenario are continu- ously changing, the computation is done on frame-by-frame basis using the overlap-add method, with frame length of 1024 samples and 75% overlap (at sampling rate of 48kHz).
A.5 DATASET DESCRIPTION
Dataset Overview. We recorded eight different subjects, four male and four female, in an acousti- cally treated room. The capture contains unidirectional conversational speech, i.e., we asked partici- pants to talk to mannequin for 15 minutes each while walking around. We collected approximately 2 hours of mono-to-binaural audio data in total. Source and listener head positions are tracked and synchronized with the recorded audio. We use the last two minutes from each subject and sep- arately recorded validation sequence as test data and kept the remaining data as training data. To the best of our knowledge, this is the first binaural data capture of its kind, i.e., modeling moving trajectories between receiver and transmitter position and recorded in regular room rather than an anechoic chamber.
Data Capture Details. The acoustic head and torso simulator is the GRAS KEMAR mannequin with the size large anthropometric pinnae inserts. Participants were free to walk around a 1.5m radius circle around the KEMAR mannequin, and prompted to cover much area as normal social conversation would. The KEMAR mannequin was wearing a B&K 4101B binaural microphone headset. The subjects wore DPA 4060-0C microphone taped next to their mouth to capture their speech. The participants wore a modified bicycle helmet with reflective markers for head-pose tracking using an OptiTrack system. Although the KEMAR mannequin did not move, KEMAR wore a headband with reflective markers for head-pose tracking for complete source/listener head- pose tracking. All tracking information was captured with a field array of 24 OptiTrack Prime 17W cameras. The audio data is recorded at 48kHz sampling rate and rigid body tracking data is collected at 120fps via motion capture software, Motive. LTC signal is used synchronize the audio recordings with OptiTrack data. The capture layout is schematically illustrated in Figure 7
A.6 EXTENDED EVALUATION
Additional Perceptual Evaluation. In order to back the results of the perceptual study in Table 4 we performed a side-by-side evaluation of our system and the DSP baseline, which was ranked be the strongest competitor to our approach. In this study, participants were presented an audio snipped rendered with DSP and the same snippet rendered with our approach. The snippets are presented side-by-side in random order to ensure an unbiased evaluation. Participants were then asked to decide which of the two methods is preferable in terms of cleanliness, spatialization, and realism. We additionally gave participants the option to select can not tell the difference as an answer. Overall, 30 participants evaluated 360 binaural snippets generated with DSP and our method, respectively. The results in Table 7 support our findings from Table our approach is preferred in terms of cleanliness and realism. For spatialization, most participants could not find a clear favorite, which is consistent with the mean opinion score of 3.75 vs. 3.76 that is reported in Table
Unseen Subjects. While previous evaluations were based on unseen audio data from speakers that part of the training set, evaluate the performance unseen speakers here. We train model in a deave--ne-speaaeer-eee
setup, i.c., we train eight models, each with another speaker being held out. Table g shows that our approach still outperforms the DSP baseline by a large margin
20

---

Published as a conference paper at ICLR 2021
(a) Side view
(b) Top view
Figure 7: (a) Side view of capture layout. (b) Top view capture layout. participant moves around KEMAR mannequin within the boundaries of marked circle. The participant speech is recorded with head mounted microphone and the binaural audio is captured with binaural microphones on the ears of the mannequin. Mannequin and participant positions are tracked with OptiTrack cameras mounted on walls of the room.
Table 7: Side-by-side study of DSP vs. our system. Participants were presented two clips, one generated with DSP, one with our approach, and were then asked to tell which one they prefer.
DSP preferred
ours preferred
can tell the difference
cleanliness
1.8%
88.9%
9.3%
spatialization realism
25.9%
27.7%
46.4%
31.5%
53. 53.7%
14.8%
on unseen subjects. This is remarkable, considering that only seven different subjects are seen during training. With a more diverse dataset, we expect the generalization quality of our approach to increase significantly.
Activation Functions. In recent works (Sitzmann et al. 2020 Tancik et al. 2020), sine activations
have been found preserve high frequency information better than other commonly used activation functions if weights are initialized appropriately. As reconstructing high frequencies is particularly important for audio modeling, we adopt this strategy in our network. Table shows a comparison of our network with ReLUs, gated convolutions as used in Van Den Oord al. (2016), and sine activations as used in Sitzmann et al. (2020). ReLUs do not perform well on audio data: their sparse outputs not well suited to model the smooth and sinusoid nature of waveforms. For this reason, WaveNet originally used gated convolutions, which we also find to work better than ReLUs in our task. Overall, however, we still find sine activations to produce the best results.
Qualitative Results. We show qualitative results on the raw waveform in Figure Note the consid-
Table 8: Generalization unseen subjects. In a saave-on-subject-ou setup, our approach still out- performs the DSP baseline by significant margin.
raw waveform
power spectrum (E2 error)
phase spectrum (angular error)
(b2 emor x
DSP
0.485
0.058
1.388
(unseen subjects)
0.265
0.058
1.099
21

---

Published as a conference paper at ICLR 2021
Table Effect of the activation function used in the temporal hyper-connollttins..
waveform (22 error 10")
power spectrum (Eg error)
phase spectrum (angular error)
ours with ReLU ours with gated convolutions ours with sine activation
0.183
0.051
0.847
0.179
0.053
0.819
0.167
0.048
0.807
source signal (monno)
right left
right
recording
recording
right
right
DSP
WaveNet
right
right
bimaural recording
biuaural recording
left
rigla
riglle
Figure 8: Qualitative results on the raw waveform. Note that 2.5D visual sound besides having an overall inaccurate waveform reconstruction fails get an accurate alignment the binaural recording. Compared to all state of the art, our approach matches the real binaural recordings best.
erable temporal shift between the mono signal captured the source's microphone and the binaural recording. A strong binauralization system is required to accurately model not only this temporal shift but also produce metrically correct waveform, iec, match the shape of the binaural record- ing's 's waveform. When comparing the results for WaveNet and the 2.5D Sound architecture, it is apparent that both approaches lack in their ability accurately match the recording's 's waveform.
22

---

Published as a conference paper at ICLR 2021
Additionally, the 2.5D sound approach fails to align its output to the recordings, This comes of no surprise as the model inherently designed to solve an upmixing problem, ie., a problem where temporal shifts do not exist. Also note that due to fı-law quantization that is typically applied in WaveNet, its results are non-smooth and introduce high-frequency noise due to quantization bins being misclassified. The DSP approach, which is to day the de-facto state of the art, performs more favorably in terms of temporal alignment and overall matching of the recording's 's waveform. Com- pared our approach, however, there are significant inaccuracies an observation that consistent with our evaluation in Section 3.3
23


Reviews:
[{"review_id": "jFPGOnYqkNZ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is about a method for synthesizing binaural audio from a mono recording of a single speaker's speech.\n\nFirst, I think the title is too general. The paper does not attempt to convert all possible sounds, but it tries to convert a single speaker's monaural speech signal to binaural audio where the speaker is moving. I think this inherent assumption is important since the method will probably not work for multiple overlapping audio sources. I suggest changing the title to \"Neural synthesis of binaural speech of a single moving speaker.\"\n\nThe first part of the network \"neural time warping\" is an interesting component that is capable of adjusting the delays conditioned on the location and orientations of the source and microphone such that a location dependent binaural audio is formed by estimating time-varying delays of the original mono recording separately for two channels. It is believable that such a module would be helpful for a single moving speaker. However, such a model would not help or work when there are more than two active audio sources. A separation module would be required for that scenario. Neural time warping is an autoregressive model which can work online. \n\nThe second stage convolutional network which uses conditioned hyper convolutions is also an interesting architecture that takes the warped signals and applies time-convolutions with kernels obtained from the conditioning input which has the time-varying locations and orientations of the source and the microphone.\n\nThe section about the loss function is also interesting in that, the time domain l2 loss is shown to not work well for accurate phase estimation, so the authors propose to add a separate phase loss term to compensate for that. I think it would be better if Figure 2 is replaced with a plot of epsilon/|yhat| versus amplitude error divided by |yhat| in (a) and versus the phase error in (b). It could be clearer than the current 2D figure which is hard to interpret.\n\nThe use of \"sine activation\" is not well justified. \"sine\" activation is useful in the first layer of a \"signal representation network\" which is different from a signal prediction network. I do not see how and why that could be helpful here.\n\nIn terms of comparisons, 2.5D method uses visual information as conditional information to generate complex masks to produce binaural audio. In this paper, visual information is replaced with the spatial conditioning information. It would help to get more information about the window size and hop size used in 2.5D since they may be an important factor that relates to the amount of delays they can introduce.  For wavenet comparison, it was not clear how the wavenet model was trained to generate binaural data. Did it use any conditioning information? If so , how? Was it applied in an auto-regressive way with randomized sampling? The wavenet audio example sounded noisy which is not typical of wavenet generated audio. It looks like the DSP method can utilize a listener specific HRTF which may be difficult to incorporate for the proposed neural model. Is it an important issue?\n\nHow does the model generalize to unseen speakers and rooms? The training and testing strategy uses the same room and the same speaker(s).  Would we have any problem when the monaural audio is recorded in some other room with some other speaker?\n\nIn Figure 8, maybe it is OK not to draw the original binaural signal for every method.\n\nIn general, I liked the neural warping and conditional convolutional components which are interesting and I liked the analysis about the loss function. The approach is an interesting way to obtain binaural version of a monaural mono speaker recording in a room. The dataset produced for the paper would also be useful for research.\n\n**Update after revision**\n\nThe revision improved the paper. Thanks for taking care of my comments.\n\nJustification of sine activations, generalization to unseen speakers experiment are nice additions.\n\nThe new title is a bit better and I think it may be OK since the goal is to perform a moving source simulation for single speech sources. Multiple speech sources can be simulated separately and added together as mentioned. \nThe authors may consider possibly a better name: \"Neural binaural synthesis from mono speech\" which emphasizes that the synthesized target is \"binaural speech\" from a single speech recording.\n\nJust a few more points.\n\n1. I think it is essential in wavenet to apply the model in an auto-regressive fashion over samples. Just using the network architecture and the loss function from wavenet is not equivalent to \"using a wavenet model\" since an essential part of the model is the autoregressive sampling which makes sure the samples are dependent and coherent. Without auto-regressive sampling, the resulting sound is poor as observed by the authors. So, I suggest to emphasize that \"autoregressive sampling\" is not performed in the paper to avoid misleading the readers.\n\n2. More explanation of 2.5D is appropriate. One wonders if using a larger STFT window size would improve its results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "c82CtsGWWA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a neural network-based model to generate binaural audio given a single-channel audio and positions of source/listener & their angles.  The authors developed a dataset of binaural audio, which will be released on acceptance.\n\nTechnical details and model architecture are available in the body of the paper, whereas additional details such as baseline DSP-based approach, proof, and dataset are available in the appendix. The model was evaluated using the dataset developed in this work.  A demo video demonstrating the capability of the model is also provided as a supplementary material.\n\nThere are a few parts need to be addressed. (1) it is unclear why DTW-based warping is required.  IIRC the warpfield here can represent not only a shift but also other monotonic & causal such as repeating.  If there is only delay between left and right, just having a shift is enough isn't it?  It is great if the authors can explain the motivation to use warpfield more clearly.  (2) The use of hyperconvolution is an interesting idea.  The equation 5 uses conditional temporal convolution.  However, audio generative models such as WaveNet uses a different architecture; gated convolution.  The gating mechanism can give additional non-linearity and so I'm wondering if you can evaluate the performance of hyperconvolution against gated convolution.  (3) too large confidence intervals in Table 4.  Although there were many evaluations, the confidence intervals were pretty large and there were overlaps among them (e.g., small overlap between DSP and \"ours\" in cleanliness, large overlaps between spatialization and realism between DSP and ours).  With this result it is difficult to claim that there was a significant improvement over the baseline system.  Please check your results and design the experiment more carefully to figure out whether there is any significant difference between them. \n Conducting side-by-side comparision is one possiblity.\n\nComments:\n-  This paper claims that it works in real time but no information about speed such as real-time factor & hardware specification are provided.\n- Sampling rate information is not explicitly provided in the experiment section.\n- 0.6 MOS difference is large, not \"a bit\".\n- Modern WaveNet models often use mixture-of-logistics (refer to Parallel WaveNet paper for details) as output rather than mu-law to achieve better quality. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "S6D8sIwAI98", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper is well written. It includes clear math notations and figures. Readers can easily follow the thought process of the authors. For example, Figure 2 shows the relation of l2 loss and phase loss with respect to target energy, indicating the importance of penalizing phase loss in the end to end system. The same observation is supported by Figure 3.\n\n2. Strong results. The proposed end2end model significantly outperforms previous SOTA in terms of objective measures and subject tests. The video demo is very convincing. The model improved spatialization and sound quality.\n\n3. High novelty. This paper proposes to impose monotonicity and causality to the learned warping function, which incorporates the physics of sound propagation. I am excited to another example of applying domain knowledge to an end-to-end model. The model includes two novel components: the neural warp network compensates the errors from geometry warp, the temporal convolution works as a post processing module to account for reverberation and other effects. Ablation study shows both components are critical.\n \n\nTo be improved:\n1. The caption for Figure 4(a) seems to be incomplete.\n\n2. It would be good to include a table to compare the proposed model with baselines in terms of model size and inference speed.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: + Interesting method for binaural synthesis from moving mono-audio
+ Nice insight into why l2 isn't the best loss for binaural reconstructions. 
+ Interesting architectural choice with nice results.
+ Nicely motivated and clearly presented idea -- especially after addressing the reviewers comments.

I agree with the idea of a title change. While I think its implied that the source is probably single source, making it explicit would make it clearer for those not working in a closely related topic. Hence, "Neural Synthesis of Binaural Speech from Mono Audio" as suggested in the review process sounds quite reasonable.

Decision: Accept (Oral)