# DIFFBAS: An Advanced Binaural Audio Synthesis Model Focusing on Binaural Differences Recovery
## 1 Introduction  
Humans can locate sound sources due to their ability to perceive differences in sound signals transferred to their left and right ears [1]. However, audio signals are usually recorded in mono channels, which cannot provide realistic spatial cues for the listener without any processing. Binaural Audio Synthesis (BAS) aims to synthesize binaural audio from mono audio based on given spatial information, such as a relative position of the sound source to the listener so that the synthesized binaural audio can provide immersive auditory experiences to the listener [2]. It is crucial, especially in the fields of virtual reality (VR) and augmented reality (AR) because it can restore real-world sound environments [3]. Most BAS methods rely on traditional digital signal processing (DSP) techniques [4] to model the true propagation process from the sound source to each ear, producing binaural audio with spatial information. As shown in Figure 0(a), the propagation modeling can be decomposed into a series of components, including HRTF (head-related transfer function), room reverberation (e.g., RIR(room impulse response)), and environmental noise, each of which is modeled as a linear time-invariant system (LTI) [5; 6; 7]. To generate binaural audio, the DSP-based BAS methods often perform a series of convolutions with these systems. Among them, HRTF represents the transmission path of sound from a certain position in space to the listener's eardrum in the free field, which is a key component for binaural audio to have a sense of space. Due to its high computational cost for accurate numerical simulation [8; 9; 10], people tend to collect general discrete HRTF databases[11; 12] in anechoic chambers. Traditional DSP methods are easy to understand and relatively easy to model mathematically. However, the actual sound propagation process has non-linear characteristics that are not appropriately modeled by LTI systems [13]. In addition, since existing HRTF databases are discrete, inaccurate interpolation algorithms [14] will cause distortion of continuously moving sound sources. Therefore, in order to synthesize more realistic and natural binaural audio, DSP-based methods are far from satisfactory. With the emergence of deep learning techniques, researchers set to utilize deep neural networks (DNN) to address the limitations of DSP-based methods. An end-to-end BAS model utilizing geometric and neural warping techniques was first proposed by [PERSON] et al. [15]. It takes mono audio and spatial information (e.g., relative position information between the sound source and listener) as inputs to produce binaural audio which correlates with people's hearing experience (see Figure 1b). The model has been trained and tested on the first and only continuous binaural dataset, comprising paired monaural and binaural audio recordings captured in the real world. Based on this dataset, considering the limited frequency distribution of training samples, [PERSON] et al. [16] rendered speech in the Fourier domain to improve the model's generalization to out-of-distribution audio. Although the introduction of DNN has brought significant advantages in addressing the nonlinear effects of sound propagation, the complexity of the sound propagation process and the supervised end-to-end training process may lead to model overfitting on small binaural datasets. In order to simplify the sound propagation process, [PERSON] et al. [17] suggested that the sound propagation process can be modeled in two stages which include a common path modeling (i.e., early sound propagation from the source to the listener's front head) and specific path modeling (i.e., the sound propagation from the listener's front head  
Figure 1: Binaural audio synthesis systems. **(a)** The DSP-based BAS model.