BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
Authors: Yichong Leng
Zehua Chen
Junliang Guo
Haohe Liu
Jiawei Chen
Xu Tan
Danilo Mandic
Lei He
Xiangyang Li
Tao Qin
sheng zhao
Tie-Yan Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Binaural audio synthesis
Audio warping
Two-stage framework
Conditional diffusion probabilistic model

Abstract: Binaural audio plays a significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This synthesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtration, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, a novel two-stage framework equipped with diffusion models to synthesize them respectively. Specifically, in the first stage, the common information of the binaural audio is generated with a single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective of two-stage synthesis with advanced generative models (i.e., the diffusion models), the proposed BinauralGrad is able to generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrad outperforms the existing baselines by a large margin in terms of both object and subject evaluation metrics (Wave L2: $0.128$ vs. $0.157$, MOS: $3.80$ vs. $3.61$). The generated audio samples\footnote{\url{https://speechresearch.github.io/binauralgrad}} and code\footnote{\url{https://github.com/microsoft/NeuralSpeech/tree/master/BinauralGrad}} are available online.

Brad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
Yichong Leng! -et Zehua Chen Junliann Guo?, Haohe Liu", Jiawei Chen", Xu Tan Danilo Mandic Lei He², Xiang- Yang Li', Tao Qin", Sheng Zhao', Tie- Yan Liu University of Science and Technology of China, Microsoft Research Asia 3 Imperial College London, Microsoot Azure Speech, University of Surrey "South China University of Technology
lyc aacc123gomaill. ust.. cn, xia.xaaanngggneeeem
wustggyyygnniiuu...oooom Sustc. edu. cn
2 {junliangguo taoon tyw.aiiiiinnililiiiiiiooorroooee
com c dht.onnnddmmmmmddiiii.nerrrr.... uk
[helei, wta.leiisszhaemmmmccooott......
http:///gthu..coo///ccorrr/////p cn
Abstract
Binaural audio plays significant role in constructing immersive augmented and virtual realities. As it is expensive to record binaural audio from the real world, synthesizing them from mono audio has attracted increasing attention. This syn- thesis process involves not only the basic physical warping of the mono audio, but also room reverberations and head/ear related filtrations, which, however, are difficult to accurately simulate in traditional digital signal processing. In this paper, we formulate the synthesis process from a different perspective by decomposing the binaural audio into common part that shared by the left and right channels as well as a specific part that differs in each channel. Accordingly, we propose BinauralGrad, novel two-stage framework equipped with diffusion models to the first stage, the common infor-
synthesize them respectively. Specifically, mation of the binaural audio is generated with single-channel diffusion model conditioned on the mono audio, based on which the binaural audio is generated by a two-channel diffusion model in the second stage. Combining this novel perspective two-stage synthesis with advanced generative models (i.e., the diffusion models), the proposed BinauralGrad is able generate accurate and high-fidelity binaural audio samples. Experiment results show that on a benchmark dataset, BinauralGrat outperforms the existing baselines by large margin in terms of both object and subject evaluation metrics (Wave L2: 0. 0.1 vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples") and code available online.
Introduction
Human brains have the ability to decode spatial properties from real-world stereophonic sounds, which helps us to locate and interact with the environment. While in artificial spaces such as augmented and virtual reality, generating accurate binaural audio from the mono is therefore essential to provide
Equua contribution.
1 This was conducted at Microsoft, Corresponding author: Tan, x
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Common Stage
Specific Stage
Figure I: An illustration of the proposed two-stage framework. x: indicates the mono audio emitted by the source object, (v', y"" indicate the binaural audio received by the left and right ears of the receiver, and y = mean(y', yy) is the mono audio calculated by averaging the two channels in The common stage models the factor affecting both the left and right channel such as the distance of source and listener and the interaction of audio with room and head. And the specific stage models the difference of left ear and right ear including the marginal distance difference and the difference of impulse response of two ears.
immersive environments for listeners. Traditional Digital Signal Processing (DSP) systems generate binaural audio by warping the mono audio according to the time difference between two cars and modeling room reverberations (e.g., room impulse response (RIR) and head/ear related filtration (e.g..
head-related transfer functions (HRTF)) through a linear time-invariant system [32 47 35) However,
they fail to generate accurate and immersive binaural audio due the non-linear nature of sound propagation and the simplification in DSP systems, such as simplifying the physical modeling of RIR and utilizing general HRTF's instead of personalized. In addition, the exact physical information of the recording environment is not available most of the time.
Instead of following the complicated synthesis process of binaural audio in traditional digital signal processing which is difficult to model, in this paper, we formulate the process from a different and novel perspective. We first decompose the synthesis of binaural audio into two steps: 1) the original mono audio is emitted by the object and then diffuses to the listener. The audio that arrives at the left and right ears are similar, as the distance between the between the object and the listener; 2) the binaural audio is then encoded by human head depending This process is personalized as the
ears is marginal compared with that
on the marginal difference of audio received at two ears 2 4 morphology of each listener is unique, which will change the internal encoding results. Precisely processing the two stages is difficult due to the complexity of physical system. Therefore, as an alternative, given that the two channels of the binaural audio is similar but with marginal differences, we propose divide the the representation of the binaural audio y = (y', yy" as two parts. One is the general part that contains the common information of two channels, which are represented as the average y/ the other one is the specific part that is distinct between the two channels encoded by human head in the end, which are denoted as 8' and 8". Then we can represent the binaural audio as y' = G + 5' and y" = ÿ + 8" respectively. The common part represents the fundamental information of the source mono audio, while the specific parts contain the difference between the left and right channels caused by room reverberance and the acoustical influence of human head.
Based on the representation, we propose a two-stage framework to model the two parts respectively. The common information of the two channels of binaural audio is generated in the first stage, and their difference are generated in the second stage. We therefore term them as first/common and second/specific stages. The common stage is encouraged to generate y while the specific stage tries to model the difference to the left and right audio. Specifically, we build our framework on denoising diffusion probabilistic models (diffusion models for short) [12], which have been shown effective and efficient to generate high-fidelty audio on the task of speech synthesis [5] 4 [7]. In the common stage, the mono audio taken as the condition of aigle-channel diffusion model, which is supervised by the average over the two channels of the golden binaural speech y. encouraging the model to generate the common information of two channels. And the specific stage, conditioned on the single-channel output of the first stage, we utilize a two-channel diffusion model to generate the audio for the left and right ears respectively. We provide an illustration of the proposed framework in Figure [

---

The main contributions of this work are summarized as follows:
We formulate the representation of binaural audio as two parts, and propose novel two-stage framework to respectively synthesize their common and specific information conditioned on the mono audio.
Equipped with denoising probabilistic diffusion models, the proposed two-stage framework is able to generate accurate and high-fidelity binaural audio.
We conduct experiments on the benchmark dataset released in [29], and the proposed framework outperforms existing baselines by a large margin, both on automatic (Wave L2:
0.128 vs. 0.157) and human evaluation (MOS: 3.80 3.61) metrics.
Related Works
2.1 Binaural Audio Synthesis
Different from text to speech synthesis that mainly generates mono speech from text [37 [3 38 binaural audio synthesis aims convert mono audio into its binaural version. Based on the physical process of sound rendering, human listening can be generally considered as source- source-medium- medium-receiver model [3]. The sound waves emitted by the object will travel through the medium, i.e., free air and then be received by listener. In medium, the sound will be diffused, reverberated and effected by the physical objects such as walls during the propagation. Room impulse response (RIR) 19 36
30 describes the distortion effect caused by surrounding environment using filters. Besides the interaction with medium, the morphology of listener (torso, head, and pinna) will also change the sound received by the
eardrum. The head-related transfer functions (HRTF) 3 6 are utilized to describe the transformation
of the sound from a direction in free air to that it arrives at the eardrum (18]. Digital Signal Processing (DSP) methods synthesize the binaural speech by combining RIR and HRTF 47 (5]]
both of which are expensive and tedious to measure, as well as missed from most open- sourced datasets in addition. Therefore, DSP methods usually utilize generic functions instead, resulting in sub-optimal generation results as the recording environments are highly specialized among different datasets.
Recently, neural rendering approaches are proposed for binaural speech synthesis. Gebru et [0] utilizing neural networks to learn HRTFs implicitly. Richard et al. [29] propose a convolution neural network based model with an additional neural time warping module to learn the time shifts from the mono to binaural audio. Some works incorporate extra modalities such as the visual information to help the synthesis binaural speech [46] 45 26). this work, we consider most basic scenario where only the mono audio is available.
2.2 Denoising Diffusion Probabilistic Models
Denoising diffusion probabilistic models (diffusion models for short) have achieved the state-of-the- art (SOTA) generation results in various tasks, including image [34
22
g
33
39
[4] and super
resolution image generation 13
31 41 (25), text-to-image generation 23
14 28 text-to-speech
synthesis [5] 27
Especially, in audio synthesis, diffusion models have shown strong ability in modelling both spectrogram features
[7
16 S and speech enhancement 200 21 42
27 and raw
waveforms 15 5.
Diffusion models require a large number of inference steps to achieve high sample quality. For high- resolution data generation, some works [8 1 28 even cascade multiple diffusion models together and achieve the SOTA quality in text-to-image generation [28]. this cascaded structure, multiple diffusion models are trained at different resolution scales, and while inference, low-resolution data samples are firstly generated with the base diffusion model, and then stacked conditional diffusion models employed for super resolution, where high-fidelity results can be achieved with moderate time cost.
In this paper, for generating binaural audio waveform with high resolution (48kHz sampling rate), we integrate diffusion models into the proposed two-stage framework, and generate the common and specific information of binaural audio with single- and two-channel diffusion models respectively.
3

---

Common Stage
Specific Stage
Diffusion Model e
N(O.1)
Diffusion Model d,
y
N(01))
y
NNO.1)
yr
(P Pa)
(Ps.Pa)
x
c
*wurp
Xgarp
y
The pipeline of the proposed framework while training. The difference between the two stages includes the condition signal c, the parameter set 0, and utilizing one-/two-chanell diffusion models in common/specific stages respectively.
Figure 2:
Methods
We introduce the proposed framework BinauralGrad in this section. We start with the preliminary knowledge about binaural audio and geometric warping, and then introduce the proposed two-stage framework well as the structure of the diffusion models.
3.1 Preliminary
Problem Definition Given the source mono audio x E R& with length N and the relative position between the source and listener p. we aim to generate the binaural audio y = (y' .v" that contains two channels of audio for left and right cars through
(w' o' (n)) [(x(n),)),
(1)
where s' and y" E RN,, and f indicates the transformation function parameterized by the proposed framework. We denote the average the binaural audio as ÿ = mean(y/ yy").
Specifically, the relative position can be described with tuple p (Ps.Pa) where p. = (Px py: P=) the spatial position that indicated by the coordinates, and Pa (gz, qy, q2, qw) is quaternions that indicates the head orientation from the listener to the source. The listener is stationary and set in the origin of the coordinate system, and therefore (Pa-Py= Py: P=) axes indicate the front, right and up directions respectively. Besides, we also denote the spatial position of the left and right ears of the listener as Plstn and Piatn respectively.
A simple way to align the temporal differences between the left and right ears is geometric warping, non-parametric method conditioned on the distance between the source and the listener:
p(n)
C [IPere(n) Platn (n)
where n indicates the current time-stamp, C is a constant that calculated by the ratio between the audio sampling rate and the speed of the sound. the predicted warpfield p(n) are usually floats, the warped signals can be computed with the linear interpolation:
Zworp (n) = (Πp(n)] p(n)) x [[(())] + (p(n) Lo(n))) #[p(n)]
(3)
where [.] and [.] indicate the ceil and floor function respectively. The warping for the left and right ears can be obtained by changing the position Pisto (n), and the warped binaural audio is denoted as (ararrps x (warp), which are with low quality without considering the diffraction of audio.
3.2 Overview of the Framework
The proposed framework contains two stages, and we train two distinct diffusion models for each stage, parameterized by 0, and 0, respectively. In the first stage, the model 0, aims to synthesize the common information of the binaural audio supervised by 7. i.e., we treat the mono average two channels the golden supervision of the common information. And in the second stage, conditioned

---

on the synthesized audio Je of the first stage, two-channel diffusion model f, tries to synthesize the golden binaural audio 3")). We provide an illustration of the proposed framework Figure We will first introduce the backbone diffusion model and then the details of two stages as below.
3.2.1 Conditional Diffusion Models
They are composed of a forward process and reverse process. In the training stage, the forward process gradually destroys the data samples into Gaussian noise with large number of time steps. At each time step f, the model learss score function that denotes the gradient information. Then, in the reverse process, with learned score functions at predefined inference noise schedules, the model can generate clean data samples from Gaussian noise in iterative denoising process.
Diffusion models are score-based generative models 12 34
The forward process converts the data samples 2g into the isotropic Gaussian noise e ~ N N I) with predefined variance schedule 0 B1 < <... Be <... < BT 1. The latent representations at time step can be directly calculated with:
g k0) = (2 v Vāzz: (1 -a))),
(4)
where the ar 1 Br, and ā I 0g can denote a corresponding noise level at time step In sampling, the reverse process starts from the isotropic Gaussian noise p(2r) ~ -V(0, f), and iteratively denoises the generated samples toward clean data 2g with the conditioning information
T pa(2o. ... - = II pa (z (1).,
(5)
Providing strong conditioning information e is usually helpful to reduce the number of inference steps and improve the generation quality. The Gaussian transition probability of each inference step is parameterized as:
po (-t-1 -
o),
1- Sg or 3r. Following previous works 12 where the variance oî is usually predefined as 1-6;
use the former one in this work. The model is trained by maximizing the variation lower bound of the likelihood p=(z)). When we parameterize the mean function as:
po(z,, c) 1/Van(z B. agco(Ze, c,)),
reweighted training objective is usually adopted in practice 12 15
(0) Ezg.e. ||e- eo + vI - ā,e, c)
(8)
For human speech signals, they usually have more energy in low-frequency region, while Gaussian noise has equal intensity at different frequencies. Hence, in the forward process, the high-frequency information is firstly destroyed, then the low-frequency information is also destroyed when the time step increases. Correspondingly, in the reverse process, the low-frequency information will be gradually generated at first, then details will be iteratively refined with following inference steps, which is suitable for accurate and high-fidelity speech waveform generation.
3.2.2 Details of Two Stages
With the understanding of the iterative refinement mechanism of diffusion models, integrate them into the proposed two-stage framework, in which a single-channel diffusion model generates the common information of binaural audio in the first stage, while a two-channel diffusion model is then used for modelling the specific information of both left ear and right ear.
In the first stage, the average ÿ of the golden binaural audio is considered as the clean data in the forward process. In this way, the diffusion model is encouraged to generate the common information of the two channels. The condition information is consistent for the forward and reverse processes. In concrete, the model takes the average of the warped binaural audio Ø = mean(s) (r worps #'garp) as the condition information (Defined in Equation 3). In addition to the mono audio and position, the
5

---

Residual Block M
Output
Conv x
Conv
Skip Connections
Conditioner
Conv
Dilated Conv
Concat
Conv x 2
Conv 2
Conv
FC
Input
Diffusion Step Encoding
Position
Conditional Audios
Figure An illustration of the model architecture. "FC" indicates the fully connected layer, "Conv" indicates the x convolution layer, and "Dilated Conv" indicates the bidirectional dilated convolution layer. The input represents the corrupted data sample 2; at the t-th diffusion step, while the position and the conditional audio have been defined in Section 3.1 functions for simplicity.
We omit the activation
condition of the first stage be denoted C1 (Ps. Pa: 2)). The position and audio information are separately processed by the model, which will be detailed introduced in Section 3.3
The second stage aims synthesizing the binaural audio through two-channel diffusion model, (y'. yy) taken as the clean data for the two channels respectively.
therefore the golden audio Different from that in the first stage, the condition information is distinct in the forward and reverse process as includes the generation results of the first stage. The average ÿ of the golden binaural audio is known in the forward process, therefore, along with the warped binaural audio, the condition information can be written as C2 = (ps.Pa, 2 Zwarp- Fwarp: I, J For the reverse process, the genera- Pa, tion results Ye of the first stage is utilized to replace the golden audio which is unknown in inference, ie., C2 = (p.. Pas
xf marpp Twarp: I, ye).
While training, the two stages are trained separately by minimizing the regression loss defined in Equation
X Butilizing different clean data and conditions. And in inference, firstly the common stage generates the mono audio Ye. conditioned on which the specific stage synthesizes the binaural audio, which are the final output of the framework. There exists an inconsistency between the training and inference of the second stage due to the different conditions, and fine-tune the second stage conditioned on the output the first stage may alleviate this problem. We leave it for future work.
3.3 Model Architecture
The model architecture of our diffusion model is shown in Figure 3 The diffusion models utilized in the two stages have similar architectures, except the channels of the input, output and condition are one and two for the common and specific stages respectively. We introduce Conditioner component to deal with the condition signals introduced in previous sections. Specifically, we utilize two separate convolution blocks to process the position as well as the conditional audio respectively, and the output features are concatenated and then processed by a final convolution layer. The conditional audio represents the concatenation of [i, xr. and [z F.papp: p: 2, Me] for the two stages. The output of the Conditioner is taken as the representation of condition signals.
The rest of our model generally follows the design in [15
The main network is based on the bidirectional variant the dilated convolution layer (24) given the non-autoregressive nature of the task. The architecture is constructed by M residual blocks, with dilated convolution layers in

---

each block. The diffusion step t is represented by positional encoding [40] and encoded by two fully connected layers before fed to the network, and then each block, the diffusion step will be further transformed by another fully connected layer that is unique among blocks, where the ouiput representation will be added to the input of the dilated convolution layer. The conditional representation modeled by the Conditioner will be added to the output of the dilated layer to bring in the conditional information. In the end, through skip connections, the outputs of residual blocks are gathered to produce the final output of the model.
4 Experiments
Dataset We utilize the dataset released in (29 as it is the largest binaural dataset captured in the wild up to now. It contains hours of mono-binaural parallel audio data 48kHz from eight different objects, recorded in a regular room. The position and orientation between the objects and the listener are tracked at 120Hz and aligned with the audio. We follow the original split of training/valid/est sets to make our results comparable.
The proposed framework is denoted as BinauralGrad, and we consider following baselines. DSP is a binaural rendering approach that simulates the spatial acoustic effects of sound sources with dynamic virtual positions. DSP utilizes RIR to model the room reverberance and HRTF to model the acoustical influence the human head. We perform RIR simulation with an open-sourced too according to the room information provided in the original paper [29]. The HRTF data comes from [9]. which was measured at a distance of 1.4 meters using the KEMAR, a manikin for acoustic testing. And we follow the procedure in to simulate the binaural sound. As the exact HRTF and RIR are not provided in the dataset, therefore the DSP results are worse than that reported in the original paper (9) WaveNet [24] where the positions of source and listener are used as condition signals and fed into a ConvNet to generate binaural audio from the mono one. WarpNet (29) which originally proposes the dataset, and stacking neural time warping module with a temporal ConvNet to generate binaural audio. For all baselines, we adjust their hyper-parameters to make the number of parameters comparable with our model ensure fair comparisons.
Baselines
Evaluation Metrics We use following metrics to evaluate the quality of synthesized binaural audio. Wave L2, which is the mean squared error between the synthesized binaural audio and the golden binaural recording. Amplitude L,2 and Phase L.2, which are the mean squared errors between the synthesized binaural speech and the binaural recording on the amplitude and phase respectively after performing Short-Time Fourier Transform (STFT) on wave. PESQ, which is the perceptual evaluation of speech quality and widely used in speech enhancement. MRSTFT, which models the multi-resolution spectral Ios:Dby by taking the spectral convergence, log magnitude loss and linear magnitude loss into considcration. Except PESQ which is the higher the better, the rest of metrics are the lower the better. Besides object metrics, we also conduct subject evaluation (Mean Option Score, MOS) to verify the audio quality intuitively.
Training Configurations As described in Section 3.3 BinauralGrad consists of M residual blocks, each of which has m = 10 bidirectional dilated convolution layers. The hidden size and the dimension of the diffusion step encoding are both set to 128. We train BinauralGrad on 8 Nvidia V100 GPUs for 1M steps, and the diffusion steps are set to 200 and 6 during training and inference respectively, mainly following previous diffusion works [15]
We will introduce the experimental results in the rest of this section. We first compare the proposed Binaural with baseline systems in terms of objective metrics, then verify the effectiveness of the proposed two-stage framework by comparing with single stage diffusion model. In addition, a subject metric MOS test is also utilized to show the intuitive performance of the model. Finally, to better understand the two-stage model, we also conduct a thorough analysis on the output of each stage.
htteer//ggtaabbboom/mmmm//////ss
http://spatialaadionnnee//zz

---

Table 1: The comparison regarding binaural audio synthesis quality. For all baselines, we report the
results based on our re-implementation.
Model
Wave L2 (x10-*)4
Amplitude L2
Phase L2
PESQ
MRSTFT
DSP
1.543
0.097
1.596
1.610
2.750
WaveNet 24
0.179
0.037
0.968
2.305
1.915
WarpNet
0.157
0.038
0.838
2.360
1.774
BinauralGrad
0.128
0.030
0.837
2.759
1.278
Table 2: The comparison with the single stage diffusion model.
Model
Wave 1.2 (x10-3) )
Amplitude 1.2.
Phase 1.2 4
PESQ
MRSTFT
Single Stage
0.144
0.031
0.877 0.837
2.544 2.759
1.269
BinauralGrad
0.128
0.030
1.278
4.1
Main Results
We report the object metrics of Binaural and other binaural speech synthesis baselines in Table The proposed framework performs consistently better than compared baselines over different metrics. Specifically, we have several observations:
Our method outperforms the main baseline WarpNet on Wave 1.2 by a large margin, showing that BinauralGrad can synthesize better binaural waveforms that are closer the golden recordings.
randomness in the waveform, the metrics the STFT results of waveform is important to evaluate the overall quality of binaural speech synthesis. Our method surpasses baseline methods by large margin in terms of Amplitude L2 and MRSTFT, and being slight better than WarpNet on Phase L2.
Since there
PESQ is widely used metric on the perceptual evaluation of speech quality. The proposed BinauralGrad achieves 2.759 PESQ score, which is significantly better than other baseline systems. The naturalness of our method is further verified by the MOS results listed in Table
conelusion, the object metrics show that the proposed two-stage framework able to generate more accurate binaural audio than existing baselines.
4.2 Comparison with Single Stage Diffusion Model
We compare BinauralGrad with a single stage diffusion model to verify the advantage of the two- stage framework. For the single stage diffusion model, its model architecture is similar to the two-channel diffusion model in the second stage of BinauralGrad, except for two differences: 1) the single stage model does not use any information from the common information, i.e., with condition = (p.: PP..Pa, _ warp' Twarp: x); 2) the single stage model is two times deeper so to make the number of parameters comparable.
The comparison of BinauralGrad with the single stage diffusion model are shown Table 2 The results show that the two-stage model performs better than the single stage including PESQ and L.2 losses of waveform, amplitude and phase, while achieving comparable MRSTFT scores, demonstrating the effectiveness of two stage framework. It is worth noting that the single stage model already outperforms the major baseline WarpNet over most metrics except Phase 1.2 as shown in Ta illustrating the stronger ability of diffusion models in synthesizing raw audio waveform than purely convolution based models.
most metrics
4.3 MOS Test
conduct three types of MOS to verify the quality synthesized audio with human evaluation, including: 1) MOS, where judges are asked to rate for the overall naturalness and fluency of synthesized audio; 2) Similarity MOS, where judges are asked to rate for the similarity between the synthesized results and the golden binaural recordings; 3) Spatial MOS, where the judges asked

---

Table 3: The MOS test results on the synthesized binaural audio. marked bold.
Best scores over baselines are
Model
MOS
Similarity MOS
Spatial MOS
Recording
3.69 = 0.24
4.52 0.23
4.01 + 0.13
DSP
3.41 : 0.23
3.97 + 0.37
3.75 + 0.15
WaveNet
3.37 = 0.20
4.14 0.29
3.41 + 0.15
WarpNet
29
3.61 = 0.24
4.25 + 0.25
3.77 + 0.13
BrnduralGrad
3.80 = 0.23
4.43 + 0.20
3.79 + 0.16
Table 4: Analysis of BinauralGrad. Stage stands for the first/common stage and stage 2 stands for second/specific stage BinauralGrad.
ID
Setting
Wave L2 (x10-3)
Amplitude L2 +
Phase L2
PESQ t
MRSTFT +
Mono
1.340
0.063
1.564
1.410
2.159
Physical Warp
0.725
0.060
1.584
1.411
2.140
Stage (ū, y))
0.300
0.043
1.070
2.000
1.682
0,200
0.031
0.598
3.031
1.020
Stage
0.128
0.030
0.837
2.759
1.278
Stage 2 on ID
0,013
0.013
0.372
4.212
0.697
rate for the sense of direction contained in the synthesized audio. For all MOS tests, the judges rate discrete score from to 5, and the higher score the better quality. The MOS results together with the 95% confidence interval are shown in Table
Firstly, the proposed BinauralGrad achieves a quite high MOS score 3.80 which not only surpasses all baselines, but also slightly outperforms the recording, illustrating the strong ability of the proposed framework in synthesizing natural audio waveforms. Secondly, the similarity MOS shows that BinauralGrad synthesize closest binaural audio to the golden binaural recording among all compared models, which is consistent with the conclusions from objective metrics. Finally, the results the spatial MOS show that DSP, WarpNet and BinauralGrad achieve similar performance on describing the sense of spatial, which might be attributed to the fact that the ahove three methods all utilize the physical warping process that brings in the interaural time difference (ITD) and results in the sense of direction
4.4 Analysis
In this section, conduct thorough analysis verify the effectiveness of each stage in framework, and try to find out the potential bottleneck that brings sparks for future work.
In each stage, the diffusion model synthesize audio conditioned on different information. Therefore, to verify the effectiveness of the model in each stage, we evaluate the performance of the condition and synthesized audio separately to illustrate the improvements brought by the model. Specifically, in addition to the synthesized audio from the two stages (ID and 5), we also calculate the performance of the physical warping results (i.e., (I x ww ID 2) and the average ÿ of the golden binaural audio. Note that as the output of the first stage and ÿ are both mono audio, we therefore calculate the scores by duplicating them to binaural such (y: y) for ID and (ye, yo) for ID 3.
The results are listed Table 4 Firstly, we can find that the performance of the physical warping (ID 2) is worse it only brings gains Wave L2 and remain consistent other metrics compared to the mono audio (ID 1). Secondly, the improvements from ID 2 to and to 5 are brought by the first and second stage of the proposed model respectively, verifying the effectiveness and necessities of both stages.
In addition, we try to explore the boundary of the framework by directly feeding the perfect condition to the second stage while inference, i.e., instead of conditioning on ye which is synthesized by the first stage, we utilize the golden average the condition which is unavailable in inference to test the best performance of the model. result, we can find that the second stage achieves an extremely good result and brings a large improvement over all metries compared with the given condition,

---

showing that the model in the second stage has been well trained. On the contrary, there still exists large margin between the synthesized result from the first stage and the label (ID vs. ID 4). In conclusion, the results in Table that the first stage is the bottleneck of the framework, which points out potential future work directions such as the end-to-end optimization of the two stages.
Conclusion
In this paper, we propose BinauralGrad, two-stage framework for binaural audio synthesis condi- tioned on mono audio. Specifically, we formulate the synthesis process from a novel perspective and divide the binaural audio into common part that is shared by two channels well distinct parts. Accordingly, single-channel diffusion model utilized to generate the common information in the first stage, conditioned on which two-channel diffusion model synthesizes the binaural audio. The proposed framework is able to synthesize accurate and high-fidelity audio samples. On a benchmark dataset, BinauralGrad achieves state-of-the-an results both in object and subject evaluation metrics. In the future, we plan to improve the training of the two stages in an end-to-end way for better performance speed up the inference of BinauralGrad, which is important for the online deployment of our method. The negative impact of our method might be the abuse, e.g. generating fake binaural audio mislead the user in some VR-based computr interactive games, resulting in physical injury.
References
[1] Niccolo Antonello, Enzo De Sena, Mare Moonen, Patrick A Naylor, and Toon Van Waterschoot. Room impulse response interpolation using sparse spatio-temporal representation of the sound field. IEÉE/ACM Transactions on Audio, Speech, and Language Processing, 25(10): 2017.
[2] Futoshi Asano, Yoiti Suzuki, and Toshio Sone. Role of spectral cues in median plane localization. The Journal of the Acoustical Society of America, 88(1):159-168, 1990.
[3] Durand R Begault and Leonard Trejo. 3-d sound for virtual reality and multimedia. Technical report, 2000.
[4] N. Chen, Y. Zhang, H. Zen, R.J. Weiss, M. Norouzi, and W. Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2021.
[5] Z. Chen, X. Tan, K. Wang, S. Pan, D. Mandic, L. He, and S. Zhao. Infergrad: Improving diffusion models for vocoder by considering inference training. In ICASSP, 2022.
[6] Corey Cheng and Gregory Wakefield. Introduction to head-related transfer functions (hrtfs): Representations of hrtfss in time, frequency, and space. In Audio Engineering Society Convention 107. Audio Engineering Society, 1999.
[7] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon. Ilvr: Conditioning method for denoising diffusion probabilistic models. In ICCV, 2021.
[8] P. Dhariwal and A. Nichol. Diffusion models beat gans image synthesis. In NeurlPS, 2021.
[9] Bill Gardner, Keith Martin, et al. Hrft measurements of kemar dummy-head microphone, 1994.
[10) Israel D Gebru, Dejan Marković, Alexander Richard, Steven Krenn, Gladstone A Butler, Fernando De la Torre, and Yaser Sheikh. Implicit hrtf modeling using temporal convolutional networks. In ICASSP. 2021.
[11] S. Gu, D. Chen, J. Bao, F. Wen, B. Zhang, D. Chen, L. Yuan, and B. Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
[12] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[13] Ho, C. Saharia, W. Chan, Fleet D. J. M. Norouzi, and T. Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
10

---

[14] G. Kim, T. Kwon, and J. C. Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
[15] Z. Kong, W. Ping, J. Huang. Zhao, and B. Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
[16] M.W.Y. Lam, J. Wang, R. Huang, D. Su, and D. Yu. Bilateral denoising diffusion models. In ICLR, 2022.
[17] S. Lee. H. Kim, C. Shin, X. Tan, C. Liu, Q. Meng, T. Qin, W. Chen, S. Yoon, and T.Y. Liu. Priorgrad: Improving conditional denoising diffusion models with data-driven adaptive prior. In ICLR, 2022.
[18] Song Li and Jürgen Peissig. Measurement of head-related transfer functions: A review. Applied Sciences, 10(14):5014 2020.
[19] Yuanqing Lin and Daniel D Lee. Bayesian regularization and nonnegative deconvolution for room impulse response estimation. IEEE Transactions on Signal Processing, 54(3):839 2006.
[20] Y. Lu, Y. Tsao, and S. Watanabe. A study speech enhancement based on diffusion probabilis- tic model. In ICASSP, 2021.
[21] Y. Lu, Z. Wang, S. Watanabe, A. Richard, C. Yu, Y. Tsao, and S. Zhao. Conditional diffusion probabilistic model for speech enhancement. In ICASSP, 2022.
[22] A. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. ICML, 2021.
[23] A. Nichol, P. Dhariwal, A. Ramesh, Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text text-guided diffusion models. arXiv preprint 12 2021.
[24] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint XXXV:16609003499 2016.
[25] K. Pandey, A. Mukherjee, P. Rai, and A. Kumar. Diffusevae: Efficient, controllable and high-fidelity generation from fow-dimensional latents. arXiv preprint arXIv:2201.00300, 2021.
[26] Kranti Kumar Parida, Siddharth Srivastava, and Gaurav Sharma. Beyond mono binaural: Gen- erating binaural audio from mono audio with depth and cross modal attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision pages 3347-3356. 2022.
[27] V. Popov, Vovk, V. Gogoryan, T. Sadekova, and M.A. Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML, 2021.
[28] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[29] Alexander Richard, Dejan Markovic, Israel D Gebru, Steven Krenn, Gladstone Alexander Butler, Fernando Torre, and Yaser Sheikh. Neural synthesis of binaural speech from mono audio. In ICLR,, 2021.
[30] Alexander Richard, Peter Dodds, and Vamsi Krishna Ithapu. Deep impulse responses: Estimat- ing and parameterizing filters with deep networks. ICASSP. 2022.
[31] C. Saharia, Ho, W. Chan, T. Salimans, D. Fleet, and M. Norouzi. Image super- supertresolution via iterative refinement. arXiv preprint 2rXiv:2104.07633 2021.
[32] Lauri Savioja, Jyri Huopaniemi, Tapio Lokki, and Ritta Väänänen. Creating interactive virtual acoustic environments. Journal of the Audio Engineering Society, 47())675--755, 1999.
[33] A. Sinha, J. Song, C. Meng, and S. Ermon. D2c: Diffusion-decoding models for few-shot conditional generation. In NewrIPS, 2021.

---

[34] Y. Song, J. Sohl-Dickstein, D.P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[35] Kaushik Sunder, Jianjun He, Ee-Leng Tan, and Woon-Seng Gan. Natural sound rendering for headphones: Integration of signal processing techniques. IEEE Signal Process. Mag., 32(2): 100-112 113, 2015.
[36] Igor Szöke, Miroslav Skácel, Ladislav Mošner, Jakub Paliesek, and Jan Černocky. Building and evaluation of real room impulse response dataset. IEEE Journal of Selected Topics in Signal Processing, 13(4):863-876, 2019.
[37] Xu Tan, Tao Qin, Frank Soong, and Tie-Yat Yan Liu. A survey on neural speech synthesis. arXiv preprint arXiv:2106. 2021.
[38] Xu Tan, Jiawei Chen, Haohe Liu, Jian Cong, Chen Zhang, Yanqing Liu, Xi Wang, Yichong Leng, Yuanhao Yi, Lei He, et al. Naturalspeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.044: 04421, 2022.
[39] A. Vahdat, K. Kreis, and J. Kautz. Lsgm: Score-based generative modeling in latent space. In NeurlPS, 2021.
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[41] C. Wang, K. Yeo, X. Jin, A. Codas, J. Klein, and B. Elmegreen. S3rp: Self-supervised super- resolution and prediction for diivection-diffusion process. arXiv preprint arXiv:2111.04639, 2021.
[42] S. Welker, J. Richter, and 'T. Gerkmann. Speech enhancement with score-based generative models in the complex stft domain. arXiv preprint vrXiv:2203.17004 2022.
[43] Frederic L Wightman and Doris J Kistler. The dominant role of low- low-freyyency interaural time differences in sound localization. The Journal of the Acoustical Society of America, 91(3): 1 1648-1661, 1992.
[44] Z. Xiao, K. Kreis, and A. Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. In ICIR, 2022.
[45] Xudong Xu, Hang Zhou, Ziwei Liu, Bo Dai, Xiaogang Wang, and Dahua Lin. Visually informed binaural audio generation without binaural audios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15485- 15494, 2021.
[46] Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang. and Ziwei Liu. Sep-stereo: Visually guided stereophonic audio generation by associating source separation. In European Conference on Computer Vision, pages 69. Springer, 2020.
[47] Dmitry N. Zotkin, Ramani Duraiswami, and Larry S. Davis. Rendering localized spatial audio in virtual auditory space. IEEE Trans. Multim., 6(4):553-566 2004.
12


Reviews:
[{"review_id": "w14dCDrWs9w", "reviewer": "Reviewer_6RDx", "summary": "This is a paper that applies the diffusion model to a binaural audio generation task. The authors use a two-stage method to accomplish their task and use a modified wavenet-like structure as their model architecture. After thorough and detailed experiments, the authors achieve the best results among the listed models, which nicely support their conclusions. The contributions of the paper include: \n- A new two-stage framework is used to generate high-quality binaural audio, with the assistance of a diffusion model.\n- The experiment results show that the proposed method achieves excellent performance on this task.\n\n\n", "questions": "- The authors mention that they use warped binaural audio as conditional input, but never explain how this audio is warped. We would like to know if these audios are warped using traditional DSP methods, or other additional methods? Does the use of such conditional inputs lead to some kind of information leakage, and have the authors performed ablation experiments on this?\n- The authors use the pattern of frequency destroying they found in the forward process of the diffusion model to explain why they use a two-stage generation method. However, in Table 2, the frequency-related evaluation metric MRSTFT of the proposed method is not as good as that of the single-stage model. Can the authors explain this phenomenon?\n- Have the authors compared the generation speed of single-stage and two-stage models and is there a significant difference between the two?\n\n\n", "limitations": "The authors make a good statement about the limitations of their work.\n", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper is easy to follow. The authors give a detailed background of the task, a clear definition of the problem, and a comprehensive explanation of the proposed method.\n- The experiments are adequate. The authors have verified the feasibility of their method through comprehensive experiments and subjective and objective evaluations, which support their conclusions very well.\n\n\nWeaknesses:\n- The values of the standard deviations for all results in Table 3 seem to be a bit too large, which indicates that there may be a large disagreement among raters about the experimental results. Such results may be unconvincing.\n\nBesides that, there are no significant weaknesses in this paper, but there are some issues that are difficult to explain and could be potential weaknesses, which will be mentioned in the next section.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W2zAtAolJZA", "reviewer": "Reviewer_YLbm", "summary": "This paper studies the problem of binaural audio synthesis, and proposes to decompose the binaural audio into a common part that shared by the left and right channels as well as a specific part that differs in each channel. A two-stage framework based on diffusion models is proposed to synthesize the two channels, respectively. Experiments on a benchmark dataset demonstrate the effectiveness of the proposed framework.", "questions": "Diffusion models are powerful and lead to good gains in terms of binaural sound synthesis results, it would be important to discuss why it is helpful and suitable for this specific task, compared to prior methods as well as its limitations. ", "limitations": "No limitations are discussed in the paper. There is only one sentence that discusses the societal impact: The negative impact of our method might be the abuse of binaural speech synthesis, which is very vague and unclear.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea to use diffusion models for this task is interesting, and the proposed two-stage framework is well-motivated. Generally, the paper is clearly written and easy to understand. Table 1 clearly shows that the proposed method based on diffusion models compares favorably to prior methods including the latest neural method WarpNet [29]. The main contribution/strength of this paper is introducing a technique---diffusion model, which is demonstrated to be powerful in many other areas, to this existing task of binaural sound synthesis, and achieves some gains.\n\nWeakness: The motivation for dividing the binaural synthesis problem into two stages can be better discussed. For example, in Figure 1, it would be useful to explicitly illustrate what information is used in the first stage and what information is used in the second stage, instead of just showing the waveforms. Moreover, diffusion models are powerful and lead to good gains in terms of binaural sound synthesis results, it would be important to discuss why it is helpful and suitable for this specific task, compared to prior methods as well as its limitations. No limitations or failure cases are currently discussed across the paper and only one sentence discusses potential societal impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0_Ejno4hgFw", "reviewer": "Reviewer_BZFH", "summary": "In this paper, the authors propose a diffusion-based system that can directly generate binaural signals. Based on analysis of the way sound propagates and humans perceive spatial impressions of sound, the authors designed a two-step system. The proposed system is evaluated by both objective measures and subjective evaluations, both of which shows that the proposed system is trained successfully. ", "questions": "L162-163: Just curious, Why? White noise has equal energies all over the frequency. \nIs it bc the test signals have, due to our auditory perception, more energy on\nthe low-frequency region? It could be useful to discuss this\n\nIn general: There are many symbols (x, y, with bars and hats etc). I found it quite confusing at the beginning and once misunderstood something was wrong. \n", "limitations": "(Mentioned in the Weakness)", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: The proposed system is designed based on a nice intuition. It’s novel, and effective. The design and result of the experiment are solid. The writing is good, too.\n\nWeakness: Directly synthesizing the binaural signals is not practical as of now given its high computation complexity. (But I do not think this is not a critical weakness)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept