BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
Authors: Susan Liang
Dejan Markovic
Israel D. Gebru
Steven Krenn
Todd Keebler
Jacob Sandakly
Frank Yu
Samuel Hassel
Chenliang Xu
Alexander Richard
Venue: ICML 2025 poster (2025)
Keywords: Spatial Audio
Binaural Speech
Generative Model
Flow Matching
Primary area: applications->language_speech_and_dialog
TL;DR: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models

Abstract: Binaural rendering aims to synthesize binaural audio that mimics natural hearing based on a mono audio and the locations of the speaker and listener.  Although many methods have been proposed to solve this problem, they struggle with rendering quality and streamable inference. Synthesizing high-quality binaural audio that is indistinguishable from real-world recordings requires precise modeling of binaural cues, room reverb, and ambient sounds. Additionally, real-world applications demand streaming inference. To address these challenges, we propose a flow matching based streaming binaural speech synthesis framework called BinauralFlow. We consider binaural rendering to be a generation problem rather than a regression problem and design a conditional flow matching model to render high-quality audio. Moreover, we design a causal U-Net architecture that estimates the current audio frame solely based on past information to tailor generative models for streaming inference. Finally, we introduce a continuous inference pipeline incorporating streaming STFT/ISTFT operations, a buffer bank, a midpoint solver, and an early skip schedule to improve rendering continuity and speed. Quantitative and qualitative evaluations demonstrate the superiority of our method over SOTA approaches. A perceptual study further reveals that our model is nearly indistinguishable from real-world recordings, with a 42% confusion rate.

BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
Susan Liang" 12 Dejan Markovic Israel D. Gebru Steven Krenn 2 Todd Keebler Jacob Sandakly"
Frank Yu² Samuel Hassel? Chenliang Xu Alexander Richard?
Abstract
1. Introduction
Binaural rendering aims to synthesize binaural au- dio that mimics natural hearing based on a mono audio and the locations of the speaker and listener. Although many methods have been proposed to solve this problem, they struggle with rendering quality and streamable inference. Synthesizing high-quality binaural audio that is indistinguish- able from world recordings requires precise modeling of binaural cues, room reverb, and am- bient sounds. Additionally, real-world applica- tions demand streaming inference. To address these challenges, we propose flow matching based streaming binaural speech synthesis frame- work called BinauralFlow. We consider binau- ral rendering to be generation problem rather than regression problem and design a condi- tional flow matching model render high-quality audio. Moreover, we design causal U-Net ar- chitecture that estimates the current audio frame soiely based on past information to tailor genera- tive models for streaming inference. Finally, we introduce a continuous inference pipeline incorpo- rating streaming STFT/ISTFT operations, buffer bank, midpoint solver, and an early skip sched- ule to improve rendering continuity and speed. Quantitative and qualitative evaluations demon- strate the superiority of our method over SOTA ap- proaches. A perceptual study further reveals that our model is nearly indistinguishable from real- world recordings, with a 42% confusion rate. We recommend that readers visit our project page for demo videos: https: htttp:://iangsusann-gg flww/
Unlike monaural audio, which conveys content in sin- gle channel with no spatial context, spatial audio presents the audience with a multi-dimensionns listening experience by rendering sounds from various directions and distances. When rendered using two audio channels and played back to the user's s ears through headphones, spatial audio also referred to as binaural audio. Its ability enhance realism and user engagement makes spatial audio key component of a wide range of immersive applications, from cinematic experiences and gaming (Raghuvanshi & Snyder, 201 8; Chaitanya et al., 2020; Broderick et al., 2018; Yadegari et al., 2024) to rapidly evolving fields such as virtual (VR), augmented (AR) and mixed realities (MR) (Zotkin et al., 2004; Kim al., 2019; Gupta et al., 2022; Schütze & Irwin- Schütze, 2018; Cohen al., 2015; Yang et al., 2020; Kailas & Tiwari, 2021; Liang et al., 2024; Huang et al., 2024).
Although a lot of work has been done in both signal pro- cessing and machine learning communities (Savioja et al., 1999; Zotkin et al., 2004; Jianjun et al., 2015; Zhang et al., 2017: Gao & Grauman, 2019; Richard et al., 2021; Leng et al., 2022; Liang et al., 2023b), the current state-of-the-art methods still struggle with achieving both (1) high-quality rendering and (2) causal and streamable inference. In par- ticular, generating high-fidelity binaural audio that is truly indistinguishable from real-world recordings, has remained an open problem. Given a (virtual) acoustic source and its audio signal, rendering binaural audio that is of such quality to deceive the listener into believing it is truly present in the space requires careful consideration and modeling of binaural cues, room reverb, and ambient noise. The poses of the sound source and receiver are key to perception. The dis- tance between them primarily affects the overall audio level, while their relative orientation influences the perceived di- rection of the sound source (e.g., interaural level and time di differences). Meanwhile, the inclusion of reverberation effects and background noise that match the environment is crucial for improving the realism and immersion of the acoustic scene. Existing approaches might not fully con- sider all of these factors, leading to suboptimal rendering performance, with noticeable differences between recorded
Work done during an internship at Meta. UUniversity of Rochester, NY, USA *Codec Avatars Lab, Meta, PA, USA. Correspondence to: Susan Liang Alexander Richard <richardalex mettacomm......m.....
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

BinauralFlow; A Causal and Streamable Approach for High-Quallt Binaural Speech Synthesis with Flow Matching Models
(real) and generated (virtual) sounds.
audio synthesis framework to render high-fidelity and continuous audio based on the mono input.
Furthermore, real-world audio rendering applications re- quire not only high-fidelity audio generation but also con- tinuous, streaming inference capability that maintains low latency, which is essential for applications where audio must be generated or processed in real time, such as live voice synthesis, interactive gaming, or augmented reality systems. However, most advanced neural rendering approaches (Gao & Grauman, 2019; Leng al., 2022; Van Den Oord al., 2016; Richter et al., 2023) do not support continuous synthe- sis, due to the non-causal model architectures and inefficient multi-step inference procedures.
We introduce a conditional flow matching approach to the binaural speech rendering problem by considering the problem from a generative perspective.
We propose a causal U-Net architecture that estimates vector fields solely based on history information. We present a continuous inference pipeline supporting the streaming inference of generative models.
We demonstrate the effectiveness of our approach, showing that our model outperforms existing SOTA approaches with a high margin. A perceptual study shows that our model is nearly indistinguishable from real-world recordings with a 42% confusion rate.
To achieve high-fidelity rendering and continuous infer- ence, we propose a flow matching- batse streaming binaural speech generation framework which we will refer as Bin- auralFlow. Predicting reverberation effects and background noise using a regression approach is challenging because these features are absent from the input audio signal and they exhibit stochastic behavior. Instead, we consider the binau- ral rendering problem to be a generative task. We design a conditional flow matching model to enhance perceptual realism by rendering realistic acoustic effects and dynamic ambient noise. To augment rendered binaural speech with precise binaural cues, we condition the model on the poses of the sound source and receiver to guide speech rendering.
2. Related Work
Our work is closely related to digital audio rendering, neural audio rendering, and generative models.
2.1. Digital Audio Rendering
Digital audio rendering approaches utilize Digital Signal Processing (DSP) techniques to render audio. These ap- proaches (Savioja et al., 1999; Zotkin et al., 2004; Jianjun et al., 2015; Zhang et al., 2017; Chen et al., 2020; 2022) estimate binaural audio with a series of linear time-invariant systems, including room impulse response (RIR) (Lin & Lee, 2006; Szöke et al., 2019; Antonello et al., 2017), head- related transfer function (HRTF) (Begault & Trejo, 2000; Cheng & Wakefield, 1999), and additive ambient noise. Due to the simplified geometrical simulation (Valimaki et al., 2012; Savioja & Svensson, 2015), non-personalized HRTFs, and the assumed stationary noise, there a noticeable qual- ity gap between real recordings and generated sounds.
Existing flow matching models typically do not support continuous inference due to non-causal model architectures and multi-step inference frameworks (Ho et al., 2020; Song et al., 2020b; Rombach al., 2022; Richter et al., 2023) commonly use a non-causal U-Net (Ronneberger et al., 2015) composed of convolution and attention blocks as backbones. Non-causal convolution kernels and the globally aware attention calculation mecha- nism break the time causality during rendering. Therefore, we introduce a causal U-Net architecture by meticulously de- signing causal 2d convolution blocks so that the prediction of the next audio chunk solely relies the past chunks.
requireeeaan requirements. Popular generative
2.2. Neural Audio Rendering
Recently, researchers have resorted to deep neural networks to render spatial audio given the powerful fitting capabilitiess of neural networks. Gao & Grauman (2019) introduce a vision- -uiieed binauralization network to generate binaural audio conditioned on a video frame. Richard et al. (2021) design a neural warp network to warp the mono audio ac- cording to the time delay and the listener position. Chen et al. (2023) and Liang et al. (2023a) utilize vision infor- mation to guide binaural audio prediction at novel poses. Although these methods achieve plausible speech results, their regression mechanism limits their generation capabil- ity, i.e., they cannot generate precise room acoustics and ambient noise that are absent from the input data.
Moreover, a causal backbone alone is not sufficient for streaming inference because of the multi-step generation process required by generative models. Starting from an initial noise, generative diffusion and flow matching models rely on an iterative denoising process which takes few steps to complete the generation process. To enable con- tinuous generation, we need to ensure time causality for all inference steps. To this end, we construct a continu- ous inference pipeline consisting of streaming STFT/ISTFT operations, a buffer bank, a midpoint solver, and an early skip schedule. In this way, we enable seamless streaming inference for U-Net-based generative models.
In summary, our contributions are:
We design a flow matching- based streaming binaural
2

---

BinauralFlow; A Causal and Streamable Approach for Iligh-Qualit, Binaural Speech Synthesis with Flow Matching Models
2.3. Generative Models
3.2. Conditional Flow Matching Models
Generative models, especially diffusion models (Ho al., 2020; Song et al., 2020a;b), exhibit strong generative capa- bilities in the audio domain (Yang et al., 2023; Liu et al., 2023; Huang et al., 2023; Kong et al., 2021; Leng et al., 2022). Based on DiftWave (Kong et al., 2021), Leng et al.
To address the quality challenge raised by the binaural audio rendering, design conditional flow matching model as an instance of the function f. We consider the binaural speech rendering problem to be a generative task and use flow matching models to generate binaural sound effects.
(2022) propose a two-stage diffusion model (BinauralGrad) to synthesize binaural audio. Richter et al. (2023) design a diffusion model for speech enhancement in the complex STFT domain (SGMSE). However, diffusion models require many sampling steps during inference, c.g., 30 steps. To reduce inference steps while maintaining performance, Lip- man et al. (2022) introduce flow matching models that simu- late the generation process with the optimal transport trans- formation. Inspired by this, we propose a flow matching- based generative framework that outperforms SGMSE with more efficient inference. We compare our work and other flow matching- based audio models (Lee et al., 2024b; Liu et al., 2024; Welker et al., 2025; Mehta et al., 2024; Du et al., 2024: Lee et al.. 2024a) in detail in the appendix.
Specifically, given an audio pair of the mono audio x and the binaural audio y, we first convert them from the time space to the time-frequency space using Short-Time Fourier Transformation (STFT): x STFT(x) C2X +1xxT and y = (y) E C2x +I)xT, where F is Discrete Fourier Transform (DFT) length, T is the number of time frames, and C represents the complex space. We repeat the mono input along the channel dimension to be two-channel so that x and y are of the same shape. Then we sample a random noise z ~ Nxx, T)) which centers around x with the radius of a. To generate the binaural audio y based on the mono input x, we aim to design a flow that moves from the source data z to the target data y.
We formulate the flow matching problem using the optimal transport formulation inspired by Lipman et al. (2022):
3. Method
In this paper, we propose BinauralFlow, a flow matching- based streaming model for binaural speech rendering. We first formulate the task in Section 3.1. To synthesize high- quality binaural audio, we introduce a conditional flow matching model that is conditioned on both pose informa- tion and mono input (Section 3.2). Then, we design a causal U-Net architecture that estimates the current chunk solely relying on the history information (Section 3.3). Finally, we present our continuous inference pipeline that improves rendering continuity and speed in Section 3.4.
dr(z) = ty + (1-t)z.
(2)
where Φe [0, 1] x (2 - C2x(+++1xxT is a time-
dependent flow function and the flow time step E [0, 1] is linear interpolation between y and z.
If we use the re- re-parameterization technique to represent 2 as x + ae, where e is a normal Gaussian noise, Φe is updated with
(L(z) = ty + (1 - f) + me) ty + (1 - l)x (1 1)oe.
(3)
3.1. Task Definition
The corresponding probability path Pt C2x(E+1)xT + Rs0 can be calculated as
[o, 1]
x
The goal of the binaural rendering task is to synthesize binaural audio (two channels one for each listener's 's ear) E 2 based on the monaural audio (one channel containing speaker's signal) x E RN,, and the poses of the speaker Pix E 7 N' and the listener Prx E R xx'''' where N is the length of an audio clip and N' is the length of a pose sequence. We represent a pose as a combination of position (ā, 7, 2) E 3 and quaternion rotation (w, x. H, 2) E IR., To solve this problem. need to learn a function f that maps the monaural audio to the binaural audio:
px(x) = NNaly + (1- (1-1)x, (1-1301))
(4)
When f = 0, po(z) is ([z)x,o²1) which is Gaussian distribution around the mono audio x with the radius of a. When gradually increases, the mean of pr(z) moves linearly from x to y and the standard deviation of (z(z) decreases. If = 1, pi (z) is (2 0). which collapses to the binaural audio y. Therefore, the flow defined in Equation (2) moves samples centered around the input audio x to the binaural audio y with gradually reduced variance.
y fx|ptxx Prx).
(1)
As mentioned in the introduction, learning of this mapping function f is non-trivial because is required to consider the binaural cues, and include the room reverb and the ambient noise, which usually are not present in the input mono signal and exhibit stochastic behavior. Moreover, of should support continuous rendering in the streaming inference setting.
Based on the definition of a flow, we can derive a time- dependent vector field 1y : [0, x (2)(F+1)xT + ( +1)xT using the following ordinary differential equa- tion (ODE):
dt 0((() = (o)((z))..
(5)
3

---

BinauralFlow: A Causal and Streamable Approach for High-Qualli Binaural Speech Synthesis with Flow Matching Models
Hidden Features it'
Generator
Complee To Roal
Causal 3D CNN Block
Cluscal 2D CNN
Causal 2D CNN Block
Causal 20 CNV Hinak
Real To
Activatt
Norm &
Gruedan Nokie
Time Pose
M Mono Audia
Fiom $, (4)
CFM Loss
Canaol DowarUpp
Causal
Up
ME.P
ctmmpl
+
RCIFF
Norm & Activati
Spectingran
Binaara Nudloy
Bisauni Spectogransy
Time
Receiver
Pose
Transuiter Poso
GT Vetter Fidld
Hidden Features h
(a) Causal U-Net Architecture
(b) Causal 2D Conv Block
Figure 1. Overview of FinwuralFlow framework. (a) shows the causal U-Net architecture. Our causal U-Net takes input the flow the (z) as well as four conditions f, Pex. Pix, and x, and outputs a predicted vector field. The U-Net consists of several Causal 2D Conv Blocks in the contracting and expanding parts. (b) displays the Causal 2D Conv Block. We design fully causal convolution, down/up-sampling and normalization layers to ensure temporal causality.
Algorithm/ 1 Training Procedure of Flow Matching Model Input: Dataset D, mono audio x, binaural audio y, trans- mitter pose Ptx. receiver pose Prx: standard deviation o, initial network ut
time step & and pass it along with other conditions into the model to predict the vector field. Finally, we calculate the CFM loss and update the model weights.
Discussion. Our conditional flow matching method shares some similarity with the simplified flow matching formu- lation (Tong et al., 2023; Jung al., 2024). However, we argue that our method is distinct from the simplified flow matching approach. (1) The simplified flow matching ap- proach injects minute perturbation (commonly 1e-4) to the flow, which almost degrades the problem to a deterministic task. Our method uses Gaussian noise of normal magnitude, maintaining the generation randomness. (2) Our method uses mono audio as important generation condition im- prove generation robustness. However, the simplified flow matching model cannot use this condition because causes model collapse. We provide experiment in Section 4.5 to validate the superiority of our approach.
while not converged do
(x,Y y: Pex, Prx] ] ~ D l/ Sample data from the dataset z ~ N(x, 13 Sample random variable
~ (0, 1) Sample time step
(z) + ty + (1 t)- (CFM (0) + lu (0)()(z), Pex: x; (y z)l 0 < Update(0, V (OLCFM(O))
end while
Output: trained network Ut
By replacing di (z) in Equation (5) with Equation (2), we calculate the vector field U,
vy (o. (2)) =
3.3. Causal U-Net Architecture
Then design deep neural network u, match the vector field ve with the conditional flow matching (CFM) LI loss:
In this section, we describe the proposed network architec- ture. To tailor the flow matching models for streaming ren- dering, we design a causal U-Net architecture that predicts the current vector field solely based on past information.
LCFM (0) EE...y wn (z), (z), Prxs Pix,
(y z)
(7)
where 0 is the learnable parameters of the deep neural net- work ut. We condition the model prediction on the poses of speaker Pex and listener Prx to accurately model the binaural clues. We also include the mono audio to provide rich sound information.
The complete network architecture is shown in Figure I (a). The input to our network is the flow d, (z) well as four conditions t, Pex. Pix. and x. and the output is the predicted vector field. Given a pair of mono and binaural audio sig- nals, x and y, we use STFT to calculate their spectrograms x and y. We sample a normal Gaussian noise of the same shape as y. We compute the flow Φ, at the time step using Equation (3). Then we concatenate de and the mono spec- trogram input to the causal U-Net. Because and Φi are complex spectrograms, we convert them to real numbers
We present pseudo code for training conditional flow matching model in Algorithm 1. We first select one sample from the dataset. Then we sample a random noise z follow- ing the Gaussian distribution and a time step t following the uniform distribution. We calculate the flow (z) at the

---

BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
Add
Datian Choret
(Left
Oligko
Buffer Bank
Streuant ng Inpux
Cheek:1
Buffee
ver Bullie
Sinaniing
Noise
Butfer Bank
Oree
Choil J
Mono Audio
Notex
Rinnural Audio
Mone
Noley
Intermediste
Rinsuml
Spectrugmam
Spectngram (b(
Spectrogram (i))
Spectrogmm
Figure 2. Continuous inference pipeline. Starting with mono audio chunk (top left, black solid-line box), compute its spectrogram via streaming STFT, noise, and duplicate channel to form noisy spectrogram (z). The trained model progressively removes the noise with a buffer bank. Finally, streaming ISTFT converts the predicted binaural spectrogram Φi (z) into binaural audio. When the next audio chunk appears (black dashed-line box), we repeat the process and synthesize scamlessly continuous binaural speech.
by considering real and imaginary parts as individual chan- nels and concatenating them along the channel dimension. Since both time step and pose vectors Ptx and Prx are low- dimensional, we employ the positional encoding technique (Vaswani, 2017; Mildenhall et al., 2021) project them into high-frequency space. We use Random Gaussian Fourier Embedding (RGFE) (Tancik et al., 2020) followed by Multi- Layer Perceptrons (MLPs) to encode these conditions. The transmitter and receiver pose features are concatenated be- fore feeding into the MLP. We inject the encoded time step and poses into the causal U-Net guide the vector field prediction. Finally, the causal U-Net estimates the vector field. We convert it back to complex space using real and imaginary channels.
ture dimension. We also add the time step and pose features with the hidden features to guide the vector field prediction. A residual path with an optional Causal Down/Up-sample Layer is included facilitate learning.
3.4. Continuous Inference Pipeline
After training BinauralFlow model, we design a continuous inference pipeline to render binaural speech streaming manner, shown in Figure 2. Given chunk of mono au- dio, apply streaming STFT operation to compute mono spectrogram. We add random noise to it and duplicate its channel to obtain the noisy spectrogram (z).. Then we use the trained model to gradually remove the injected noise. The denoising process involves several steps and we design buffer bank store the buffer of each step. When the next chunk is fed, we retrieve the buffer according the time step from the buffer bank and reload it the model. We leverage midpoint solver and early skip schedule to improve the denoising speed. Finally, we apply streaming ISTFT to convert the predicted binaural spectrogram Φ1 (z) to binaural audio. When the next audio chunk appears, we repeat the process and generate continuous binaural speech. Below we describe the individual components that enable continuous inference and improve rendering speed.
Causal U-Net has a contracting part and an expanding part with skip connections between them. Each part consists of several Causal 2D CNN blocks, with architecture shown in Figure (b). Each block contains Norm and Activate layers, Causal Convolution layers, and optional Causal Down/Up- sampling layers. In the Norm and Activate layer, utilize GroupNorm (Wu & He, 2018) 8) to stabilize training but we limit the computation to each individual frame rather than all frames to ensure causality. We apply the Sigmoid Linear Unit (SiLU) (Hendrycks & Gimpel, 2016) as an activation function. The Causal Convolution layer is 3x3 convolu- tion layer with a stride of size and one-side padding of size 2. One-side padding restricts the receptive field of the convolution kernel to the historical information. Because U-Net requires reducing or increasing the feature dimension in each block, we design a Causal Down/Up-samplin layer. The Causal Downsample layer contains 4x4 convolution function with stride of size 2, which reduces the feature dimension by half. The Causal Upsample layer contains 4x4 transposed convolution function, which doubles the fea-
Streaming STFT / ISTFT. We adapt STFT and ISTFT for streaming processing by adding buffers and adjusting the padding manner. We prepend the buffer content to each chunk and update the buffer with the end of the chunk.
Buffer Bank. In the causal U-Net, introduce buffers each causal convolution layer to the hidden features of the current audio chunk. These buffers then used to pad the next audio chunk. Since the denoising process involves multiple inference steps, reusing the same buffer
5

---

BinauralFlow; A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
across all steps would overwrite historical information. To address this, we construct dictionary- buffer bank B {B, ) to store network buffers of all time steps t. During inference, at time step i, we retrieve corresponding buffers B, from the buffer bank. The network huffers B, are loaded into the U-Net to complete the vector field prediction. Afterword, we store the updated buffers back to the buffer bank to replace B, We repeat this process until reaches 1.
Standard Schedule
0
0.5
Early Skip
0
0.5
Late Skip
0
0.5
(a) Different Time Schedules
Midpoint Solver. The inference process requires solving the following ODE to obtain Φ1 (z):
GT Audio
Stundard Schedule
Early Skip
Late Skip
Left Left Chaned
Mono Andio
d
dt 0r(z)) (z((r(z))))
<n(()) 2.
Right Hight Chamall
where we omit other model inputs for simplicity. Among different numeric solvers, we choose the Midpoint solver because it effectively reduces the number of function eval- uations while maintaining the performance (Lipman et al., 2022). We present pscudo code of utilizing the Midpoint solver to solve the ODE in the appendix.
(b) Generated Binaural Spectrograms with Different Time Schedules
Figure 3. The early skip time schedule. The use of an early skip strategy effectively reduces the inference steps and retains the generation performance.
Early Skip Schedule. To further reduce the number of function evaluations, we propose an early skip schedule. A standard time schedule divides the interval from 0 to into equal segments and moves sequentially from 0 to 1. As shown in Figure 3 (a), we design two new schedules: an early skip schedule that skips the first half segments and a late skip schedule that avoids the second half segments. We empirically observe that the use of the early skip sched- ule does not compromise rendering quality while the late skip degrades the performance, with worse modeling of the background noise (see Figure 3 (b)). We speculate that flow matching may be able to correct the errors from the first half during the second half of inference, so even if we conduct early skipping, it does not noticeably affect performance. Therefore, we utilize the early skip strategy to reduce the inference steps to 6. In comparison, SGMSE model (Richter al., 2023) generates comparable results with 30 steps.
the head while sitting a chair. We split the dataset into "raining/vllidationneest subsets with 8.47/0.86/1.33 hours of each subset. The test subset contains two additional speak- ers, male and female, not seen during training. See the appendix for details on the data collection setup.
Baselines. We compare our approach with digital audio rendering approaches and more advanced neural audio ren- dering approaches. We choose SoundSpaces 2.0 (Chen et al., 2022) as a DSP baseline given its powerful spatial audio ren- dering capability. For neural audio rendering models, we uti- lize 2.5D Visual Sound (Gao & Grauman, 2019), WaveNet (Van Den Oord et al., 2016), and WarpNet (Richard et al., 2021) as regression- based baselines, and use BinauralGrad (Leng et al., 2022) and SGMSE (Richter et al., 2023) as generative baselines. BinauralGrad is the -ate-of-theeart approach for the binaural speech synthesis task, which is a two-stage diffusion model.
4. Experiments
Metrics. For quantitative evaluation, we leverage three metries following WarpNet (Richard al., 2021) and Bin- auraIGrad (Leng et al., 2022): waveform L2 error (L2), magnitude L2 error (Mag), and phase angular error (Phase).
4.1. Experiment Details
Dataset. To evaluate BinauralFlow, we collect a new high- quality binaural dataset. We record 10 hours of paired mono and binaural data at 48 kHz along with the head poses of the speaker and the listener. To match real-world scenarios, we collect data in standard room without significant sound- proofing or sound-absorbing materials. The background noise from multiple AC vents and electronic equipment is recorded. Furthermore, instead of using binaural man- nequins and loudspeakers, both the speaker and the listener are real participants. During recording, the speaker is free to move anywhere in the room, and the listener free to move
4.2. Quantitative Comparison
We compare our method with existing baselines includ- ing the state-of-the-an approach BinauralGrad (Leng et al., 2022). We present the metric results in Table 1, where lower values mean better performance. We show the number of function evaluations (NFE), i.e., how many times the model is called during binaural speech synthesis, and the model type for each method. As shown in the table, DSP and
6

---

BinauralFlow; A Causal and Streamable Approach for High-Qualli Binaural Speech Synthesis with Flow Matching Models
Table Quantitative comparison with existing baselines. We show the model type (R: Regression and G: Generation), the numher of
function evaluations (NFE), the inference speed, and the model size of each approach. The L2 error is on the scale of 1e-5.
Methods
Type
NFE
Speed (ms)
Model Size (MB)
1.2
Mag +
Phase 4
SoundSpaces 2.0 (Chen et al., 2022)
1
4.91
0.0129
1.58
2.5D Visual Sound (Gao & Grauman, 2019)
1
1.1
82.0
2.78
0.0174
1.56
WaveNet (Van Den Oord et al., 2016)
R
I
21.0
32.7
2.79
0.0175
1.57
WarpNet (Richard et al., 2021)
R
1
21.9
32.8
2.79
0.0176
1.57
BinauralGrad (Leng ct al., 2022)
G
6
221.1
52.9
2.93
0.0143
1.33
SGMSE (Richter et al., 2023)
30
770.2
273.6
1.55
0.0076
1.43
BinauralFlov (Ours)
G
6
163.0
314.5
1.00
0.0071
1.33
BinauralGrad
SGMSE
1 Ours
012
0,15
Mag
0.005
NFE
NFE
NFE
Figure 4. Performance with respect to the NFE. We evaluate all generative models using the same NFE for a fair comparison.
0.1
In Table 1, the default NFE of different generative models as recommended by the authors. We further evaluate these generative models using the same NFE for more fair comparison. We test all models with 6, 30, 60 NFEs and report the results in Figure 4, where each subfigure displays results for one metric. As shown in the figure, our approach consistently outperforms other generative models across different NFEs, especially in the 1.2 and Mag metrics.
Moro Audio 0.-
0.1
SoundSpeccs
-0.
www.
SGMSE 0.5
0.1
4.3. Qualitative Comparison
0.
To provide an intuitive comparison between different mod- els, we display the waveforms of rendered binaural speech of various methods in Figure 5. The first row is the mono audio, the last row is the recorded audio, and the audios predicted by different methods are between them. The SoundSpaces approach estimates an inaccurate time delay between the transmitted mono audio and the received binaural audio. BinauralGrad and SGMSE predict accurate time delay but their amplitudes are mismatched. In comparison, ralFlow model correctly predicts the time delay and audio amplitude. We show more results in the appendix.
0.1
Audo
Figure 5. Qualitative comparison between different baselines. We display waveforms of rendered spatial audio.
regression-based models underperform the generation- models. Compared with BinauralGrad, SGMSE exhibits better generation quality in terms of L2 error and Mag er- ror, but falls short in the Phase error. Our BinauralFlow model consistently outperforms all baselines with consider- able margins. We also include the inference speed and the model size of each approach. We test the inference speed on single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 683 ms. Our model achieves the fastest inference speed among generative models. These results demonstrate that our model achieves a more favorable trade- off between performance and inference speed compared to the baseline approaches.
Binau-
4.4. Perceptual Study
We conduct a comprehensive perceptual evaluation to assess the quality and realism of rendered outputs. When dealing with questions of the realism of generated samples, per- ceptual study is a more important indicator than numerical analysis because humans can perceive the authenticity of speech, and are sensitive to subtle but unnatural variations in sound, which is difficult capture using purely numeri-

---

BinauralFlow; A Causal and Streamable Approach for High-Qualli Binaural Speech Synthesis with Flow Matching Models
Table Perceptual study. We report results of ABX, AB, and MUSHRA evaluation tasks, where higher values indicate better realism.
Methods
NFE
ABX CR 1
A-B CR t
Environment Score Spatialization Score t
0% difference GT)
50% (random
different from reference(i1)
(identical to meterencoO
SoundSpaces 2.0 (Chen ct al., 2022)
5%
12%
BinauralGrad (Leng et al., 2022)
4%
3%
SGMSE (Richter et al., 2023)
30
11%
21%
41.1 + 23,0
57.51 29.5
BinauralFlow (Ours)
30%
42%
68.4 + 23.4
83.1 + 18.9
Ground Truth
87.4 + 13.5
89.9 + 11.1
cal metrics. We perform the study in quiet, acoustically treated room, with carefully calibrated playback levels and equalized headphones. See appendix for more details.
Table 3. Performance comparison hetween different flow matching approaches. The 1.2 error is on the scale 1e-5. Methods
L21
Magl
Phuse]
We recruit a total of 23 participants and request them to complete the following tasks:
Simplified Flow Matching (Tong et al., 2023)
1.86
0.0101
1.35
Binaural (Ours)
1.00
0.0071
1.33
ABX test: subjects presented with 3 tracks, A, B, and X, and asked if X is A or if X is B (X is always one of them, and either A or B is the ground truth). This task measures if there is a perceivable difference between generated and recorded (ground truth) sounds. A-B test: subjects are presented with and B and asked which they think real recording (one is al- ways the ground truth). The task measures if users can reliably identify generated versus real sounds.
GTT Audio
Continuoes Inference Piecline
MUSHRA evaluation: subjects are presented with reference (ground truth) and generated samples, and asked to rate their similarity in terms environment (ambient noise and reverberation) and spatialization (sound source position). Scores range from 100, with higher scores indicating greater similarity.
Non-strcaming Inforence Pipeline
Bight Chana!
Figure 6. Output spectrograms using different inference pipelines.
For the ABX and A-B tests, we define a confusion rate metric (CR) that calculates how often users confuse the rendered sound with the recorded one and make a wrong choice. The maximum value of a confusion rate 50%, i.c. users cannot distinguish sounds and make random decisions.
achieves lower L2, Mag, and Phase errors, showing the effectiveness of our conditional flow matching approach.
Continuous Inference Pipeline. We compare our con- tinuous inference pipeline and the non-streaming infer- ence pipeline and show the generated spectrograms Fig- ure 6. Given sequence of audio chunks, the non-streaming pipeline binauralizes each chunk individually, causing no- ticeable artifacts between adjacent chunks. In contrast, our pipeline synthesizes seamlessly smooth spectrograms.
We show the perceptual evaluation results in Table 2. For all tasks, our approach outperforms other approaches with noticeable margins, showing remarkable rendering realism. In particular, in the A-B test we achieve a CR of 42% (the upper bound is 50%), showing that users can barely distin- guish our generated sounds from the recorded samples.
Real-Time Factor: We calculate the real-time factor of our model for different numbers of function evaluations on a single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 0.683 seconds. As shown in Table when NFE is set to 6, the real-time factor is 0.239. If we sacrifice some performance for faster inference, setting NFE to results in an RIF of 0.04. Our model demonstrates potential for real-time streaming generation.
4.5. Performance Analysis
We analyze the impacts different design choices on our binaural speech synthesis framework.
Flow Matching Methods. In Section 3.2, we discuss the difference between proposed flow matching model and the simplified flow matching framework in (Tong al., 2023). Comparison results are shown in Table 3. Our method
Data Scale. Recording 10 hours of data in real-world
8

---

BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models
and continuous inference pipeline. Our framework passes existing baselines with significant improvement both quantitatively and qualitatively. A comprehensive percep- tual study demonstrates that our model synthesizes binaural speech that is nearly indistigguishable from real recordings.
Table Real-time factor of our BinauralFlow model. We test the inference speed with different NFEs single 4090 GPU.
NFE
Inference Time (sec)
Real-Timee Time Factor
0.027
0.040
2
0.055
0.081
4
0.109
0.160
0.163
0.239
0.217
0.318
10
0.271
0.397
Phetraiing
Preizaining
Zero-Shot
6006
6014-
12
8:000
144
8:000
1.25
25
75
Usage
Usage
Usage
Figure 7. Large-scale pretraining strategy. We propose pretraining our model using massive data to improve data efficiency and en- hance generalization in downstream tasks.
scenarios is costly and labor-intensive. To understand how data quantity affects our model's performance, we evaluate it using different amounts of training data (1%, 5%, 10%, 25%, 50%, 75%). The results, shown in Fig- 7 (orange line), reveal significant performance decline when using less than 25% of the data.
To address this limitation, we develop a large-scale pre- training strategy using loudspeakers and artificial binaural heads instead of real individuals. While the use of artificial heads and loudspeakers reduces the quality and authentic- ity of the binaural data, it allows dataset with over 7, 700 hours of binaural audio data, encom- passing 97 speaker identities from the English multi-speaker VCTK corpus (Yamagishi et al., 2019) played by the loud- speaker. See the appendix for more details about the sytem and capture setup.
capture large-scale
We pretrain our BinauralFlow model on this dataset before fine-tuning it with limited real human data. As shown in Figure 7 (blue lines), this pretraining strategy significantly improves performance. The pretrained model's zero-shot performance (red stars) matches or exceeds that of model trained from scratch using only 1% or 5% real data. This demonstrates our model's robust generalization capabilities and its potential for various applications.
5. Conclusion
In this paper, we propose BinauralFlow, a streaming flow matching framework, that achieves high-quality continuous binaural speech rendering. Our framework consists of con- ditional flow matching model, a causal U-Net architecture,

---

BinauralFlow; A Causal and Streamable Approach for High-Quallt, Binaural Speech Synthesis with Flow Matching Models
Impact Statement
Cheng, C. and Wakefield, G. H. Introduction head- related transfer functions (hrtfs): Representations of hrtfss in time, frequency, and space. In Audio Engineering So- ciery Convention 107. Audio Engineering Society, 1999.
Our work is designed to improve the rendering quality of binaural speech synthesis. is not designed to modify the content of the input mono signal, but only to spatialize it, Le., place the source within an acoustic environment. How- ever, we acknowledge that the enhanced realism may raise concerns about potential misuse, such as the creation of highly realistic deepfake audio. To address these risks, we emphasize the importance of adhering to ethical guidelines, fostering transparency in applications, and promoting re- sponsible use of the proposed methods. Additionally, future research should focus on developing robust mechanisms for detecting and preventing misuse.
Cohen, M., Villegas, J., and Barfield, W. Special issue on spatial sound in virtual, augmented, and mixed-reality environments. Virtual Reality, 19:147-148, 2015.
Du, Z., Wang, Y., Chen, Q. Shi, X., Lv, X., Zhao, T. Gao, Z., Yang, Y., Gao, C., Wang, H., et al. Cosyvoice Scalable streaming speech synthesis with large language models. arXiv preprint arXiv:2412.10111 20244
Gao, R. and Grauman, K. 2.5 d visual sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and 2019.
References
Pattern Recognition, pp.
Antonello, N., De Sena, E.. Moonen, M., Naylor, P. A., and Van Waterschoot, T. Room impulse response interpola- tion using a sparse spatio-temporal representation of the sound field. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(10):1929-194) 2017.
Gupta, R., He, J., Ranjan, R., Gan, W.-S., Klein, F., Schneiderwind, C., Neidhardt, A., Brandenburg, K., and Välimäki, V. Augmented/mixed reality audio for hear- ables: Sensing, control, and rendering. IEEE Signal Processing Magazine, 39(3):63-888 89, 2022.
Begault, R. and Trejo, L. J. 3-d sound for virtual reality and multimedia. Technical report, 2000.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). arXiv preprint arXiv: I 600.08845, 2016.
Broderick, J., Duggan, J., and Redfern, S. The importance of spatial audio in modern games and virtual environments. In 2018 IEEE Games, Entertainment, Media Conference (GEM), pp. 1-9. IEEE. 2018.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Chaitanya, C. R. A., Raghuvanshi, N., Godin, K. W., Zhang, Z., Nowrouzezabraii D., and Snyder, J. M. Directional sources and listeners in interactive sound propagation using reciprocal wave field coding. ACM TOG, 39(4): 44 2020.
Huang, C., Marković, D., Xu, C., and Richard, A. Modeling and driving human body soundfields through acoustic primitives. In European Conference on Computer Vision, pp. 1-17. Springer, 2024.
Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z. Make-an-audie Text-to-audio generation with prompt-enhanced diffusion models. In ICML, pp. 13916-13932, 2023.
Chen, C., Jain, U.. Schissler, C., Gari, S. V. A., Al- Halah, Z., Ithapu, V. K., Robinson, P., and Grauman, K. Soundspaces: Audio-visual navigation in 3d environ- ments. In ECCV, pp. 17-36, 2020.
Jianjun, H., Tan, E. L., Gan, W.S.,, al. Natural sound rendering for headphones: integration of signal process- ing techniques. IEEE Signal Processing Magazine, 32(2): 100-113 13, 2015.
Chen, C., Schissler, C., Garg, S., Kobernik, P., Clegg, A., Calamia, P., Batra, D., Robinson, P., and Grauman, K. Soundspaces 2.0: A simulation platform for visual- acoustic learning. Advances Neural Information Pro- cessing Systems, 35:8896 8911, 2022.
Jung, C., Lee, S., Kim, J.-H., and Chung, S. Flowavse: Efficient audio-visual speech enhancement with condi- tional flow matching. arXiv preprint M.Xiv:24060092880 2024.
Chen, C., Richard, A., Shapovalov, R., Ithapu, K., Neverova, N., Grauman, K., and Vedaldi, A. Novel- view acoustic synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6409-660 0419966499222 2023.
Kailas, G. and Tiwari, N. Design for immersive experience:
Role of spatial audio in extended reality applications. In Design for Tomorrow-Volume 2: Proceedings of ICoRD 2021, pp. 853-863. Springer, 2021.
Chen, Y., Niu, Z., Ma. Z., Deng, K., Wang, C., Zhao, J., Yu, K., and Chen, X. F5-tts: fairytaler that fakes fluent and faithful speech with flow matching. arXiv preprint arXiv:24 1..:22400.68855 2024.
Kim, H., Remaggi, L., Jackson, P. J., and Hilton, A. Im- mersive spatial audio reproduction for vr/ar using room
10

---

BinauralFlow; A Causal and Streamable Approach for High-Quallt, Binaural Speech Synthesis with Flow Matching Models
acoustic modelling from 360 images. In 2019 IEEE Con- ference Virtual Reality and 3D User Interfaces (VR), pp. 120-126. IEEE, 2019.
Mehta, S., Tu, R., Beskow, J., Székely, É., and Henter, G. E. Matcha-tts: A fast tts architecture with conditional flow matching. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1 11341-11345. IEEE, 2024.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1):99-106, 2021.
Kong, Z., Ping, W. Huang. J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, perf-eerformance deep learning library. Advances in neural information processing systems, 32, 2019.
Lee, S.-H., Choi, H.-Y., and Lee, S.-W. Accelerating high-fidelity waveform generation via adversarial flow matching optimization. arXiv preprint arXiv:2408.08019, 2024a.
Lee, S.-H., Choi, H.-Y., and Lee, S.-W. Periodwave: Multi- period flow matching for high-fidelity waveform genera- tion. arXiv preprint 24xi:2408.00544: 2024b.
Raghuvanshi, N. and Snyder, J. Parametric directional cod- ing for precomputed sound propagation. ACM TOG, 37 (4): (4):1-14,20 2018.
Leng, Y., Chen, Z., Guo, J., Liu, H., Chen, J., Tan, X., Mandic, D., He, L., Li, X., Qin, T., et al. Binauralgrad: A two-stage conditional diffusion probabilistic model for binaural audio synthesis. Advances Neural Information Processing Systems, 35:23689-2233 23700, 2022.
Richard, A., Markovic, D., Gebru, D.,, Krenn, S., Butler, G. A., Torre, F,, and Sheikh, Y. Neural synthesis of binau- ral speech from mono audio. In International Conference on Leurning Representations, 2021.
Liang, S., Huang, C., Tian, Y., Kumar, A., and Xu, C. Av- nerf: Learning neural fields for real-world audio-visual scene synthesis. Advances in Neural Information Pro- cessing Systems, 36:37472-37490 2023a.
Richter, J., Welker, S., Lemercier, J.-M., Lay, B., and Gerk- mann, T. Speech enhancement and dereverheration with diffusion basen generative models. IEEE/ACM Transac- lions on Audio, Speech, and Language Processing, 31: 2351-2366 2023.
Liang, S., Huang, C., Tian, Y., Kumar, A., and Xu, C. Neural acoustic context field: Rendering realistic room impulse response with neural fields. arXiv preprint MIXi::2309.15977, 2023b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference on computer vision and patter recognition, pp. 10684-10695, 2022.
Liang, S., Huang, C., Tian, Y., Kumar, A., and Xu, C. Language-guided joint aisio-visual editing via one-shot adaptation. In Proceedings of the Asian Conference Computer Vision, pp. 1011-1027, 2024.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assisted intervention 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pp. 234-241. Springer, 2015.
Lin, Y. and Lee, D. D. Bayesian regularization and nonnega- tive deconvolution for room impulse response estimation. IEEE Transactions on Signal Processing, 54(3):839-847, 2006.
Lipman, Y., Chen, R. T., Le, M. Flow matching for generative modeling. arXiv preprint ar Xxxx:22100002444 2022.
H., Nickel, M., and
Savioja, I. and Svensson, U. P. Overview of geometrical room acoustic modeling techniques. The Journal of the Acoustical Society of America, 138(2):708-733) 2015.
Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D. P., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. In ICML, pp- 21450-21474, 2023.
Savioja, L., Huopaniemi, J., Lokki, T., and Väänänen, Creating interactive virtual acoustic environments. Jour- nal of the Audio Engineering Society, 47(9):675-705. 1999.
Liu, P., Dai, D., and Wu, Z. Rfwave: Multi-band rectified flow for audio waveform reconstruction. arXiv preprint arXiv:2403.00 050 2024.
Schütze, S. and Irwin-Schütze, A. New Realities in Audio: A Practical Guide for VR. AR, MR and 360 Video. CRC Press, 2018.

---

BinauralFlow: A Causal and Streamable Approach for High-Qualin Binaural Speech Synthesis with Flow Matching Models
Song, J., Meng, C., and Ermon, S. Denoising diffusion im- plicit models. arXiv preprint arXiv:2010.02502 2020a. 2020a.
Yang, D., Yu, J., Wang, H., Wang, W., Weng. C., Zou, Y., and Yu, D. Diffsound: Discrete diffusion model for text- to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020b.
Yang. J., Sasikumar, P., Bai, H., Barde, A., Sörös, G., and Billinghurst, M. The effects of spatial auditory and visual cues mixed reality remote collaboration. Journal on Multimodal User Interfaces, (4(4):3773352 2020.
Szöke, I., Skácel, M., Mošner, L., Paliesck, J., and Černocký, J. Building and evaluation of real room impulse sponse dataset. IEEE Journal of Selected Topics in Signal Processing, 13(4))863-88) 876, 2019.
Zhang, W. Samarasinghe, P. N., Chen, H., and Abhayapala, T. D. Surround by sound: A review of spatial audio recording and reproduction. Applied Sciences, 7(5):532, 2017.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier features let networks learn high fre- quency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Zhang, Y., Liang, S., Yang, S., Liu, X., Wu, Z.. Shan, S., and Chen, X. Unicon: Unified context network for robust active speaker detection. In Proceedings of the 29th ACM inte international conference on multimedia, pp. 3964-3972, 2021.
Tong, A., Malkin, N., Huguet, G., Zhang, Y., Rector-Brooks, J., Fatras, K., Wolf, G., and Bengio, Y. Improving and generalizing flow-based generative models with mini- batch optimal transport. arXiv preprint arXiv:2302.00482. 2023.
Zotkin, N., Duraiswami, R., and Davis, L. Rendering localized spatial audio in virtual auditory space. IEEE Transactions on multimedia, 6(4):553-564, 2004.
Valimaki, V.. Parker, J. D., Savioja, L., Smith, J. O., and Abel, J. S. Fifty years of artificial reverberation. IEEE Transactions on Audio, Speech, and Language Process- ing, 2025): 2012.
Van Den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalehbrenner, N., Senior, A., Kavukcuoglu, K., et al. Wavenet: A generative model for raw audio. arXiv preprint (att::6600039999 12, 2016.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Welker, S., Le, M., Chen, R. T., Hsu, W.-N., Gerkmann, T., Richard, A., and Wu, Y.-. Flowdec: A flow-based full- band general audio codec with high perceptual quality. arXiv preprint arXi::2503.01885, 2025.
Wu, Y. and He, K. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018
Yadegari, S., Burnett, J., Kestler, G., and Pisha, L. Spa- tial audio and sound design in the context of games and multimedia. In Encyclopedia of Computer Graphics and Games, pp. 1714-1721. Springer, 2024.
Yamagishi, J., Veaux, C., and MacDonal,, K. CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92), 2019. URL https:// doi. doi.org/10. 7488/ ds28//ds/655...
12

---

BinauralFlow: A Causal and Streamable Approach for High-Qualli Binaural Speech Synthesis with Flow Matching Models
A. Demo Videos
To help understand our work, we have created several demo videos showcasing BinauralFlow's s binaural speech rendering capability. We include these demo videos on our webpage. We highly recommend that readers watch these videos to gain a deeper understanding of our research. In each video, we show a top-down view of the room along with the poses of the speaker and the listener. The speaker is denoted "Tx" and the speaker's trajectory is shown in blue. The listener is denoted as "Rx" and the listener's trajectory is shown in red.
In the directory of each sample, we include two subdirectories: "Comparison" and "Flip_Test". In the "Comparison" subdi- rectory, we display the results of SoundSpaces ("dsp"), BinauralGrad ("bgrad")) SGMSE ("sgmse"), ) and our BinauralFlow ("ours"). We also include the mono input ("mono") and the recorded binaural audio ("gnd"). In the "Flip_Test" subdirectory, we compare the synthesized sound and the ground-truth sound by using a flip-test technique. We periodically flip the sound between the synthesized sound and the ground-truth speech every seconds.
B. Implementation Details
We implement our streaming flow matching model with the PyTorch framework (Paszke et al., 2019). Our U-Net consists of seven Causal 2D Conv Blocks for the contracting and expanding parts. We only conduct the downsampling and upsampling operations four times. We set the window length as 512, the hop length as 128, and use a Hann window when applying STFT. The input audio length is 32768 and the spectrogram is of shape 256 x 257. We use the Adam optimizer (Kingma, 2014) with a learning rate of le-4 and a weight decay rate of 1e-5. We set the standard deviation of z as 0.5. We use 6 steps to solve the ODE with the midpoint solver and an early skip schedule.
C. Midpoint Solver
We present pseudo code of the inference process using the midpoint solver Algorithm 2.
Algorithm 2 Inference Procedure with Midpoint Solver
Input: Trained network u,, mono spectrogram inference steps
~ N(xx 12) // Sumple random variable
++ 0
(z) + z
- 2/n
while < do
o'
x- (((e(z)) Pix: Pix, x: 0) // Calculate vector
Φv t (z) + v'8 1/ Calculate flow +
x" - (rbe (Z), (2). Prx. Pix, 0) // Calculate vector field +
e == (v' ('))
Average the vector field
(z(z) + (e()z) + vō // Update the llww
1 1-+ Update the time step
end while
y + (z(z)
Output: binaural spectrogram y
Given a trained network ny a mono spectrogram x, and a predefined inference step n, we sample a random noise z following the Gaussian distribution N (xxx o²I). We initialize some variables, including f, dx (z), and 5. For each time step t, we calculate the vector field at two places, I and + 6. Then we average these two vector fields and update the flow using the average vector field. In the end, we output updated di (z) as the rendered binaural spectrogram y.
D. Data Collection Setup
The data collection featured a seated listener (they were free to move their head), and a speaker talking within approximately I m radius from the listener. single participant acted as the listener, while three participants were captured as speakers, with one participant used as a part of the training set. The captures were performed a non-anechoie room. The audio system featured calibrated B+K 4101-B Binaural Microphone pair worn by the listener, as well as several DPA 4060s microphones
13

---

BinauralFlow: A Causal and Streamable Approach for High-Qualin Binaural Speech Synthesis with Flow Matching Models
(a) We collect data a standard room without significant soundproofing or sound-absorbing materials. The background noise from multiple air conditioning vents and electronic equipment is recorded.
(b) We perform the perceptual study in a quiet, acoustically treated room, with carefully calibrated playback levels and equalized headphones.
Figure 8. Capture and evaluation setups.
mounted to a VR headset worn by the speaker, with guaranteed phase synchronization. Poses of both the speaker and listener were recorded via an OptiTrack tracking system. The speaker was tracked with IR-reflective markers mounted on the headset, and the listener was tracked via small facial IR-reflective markers. The setup is shown in Figure 8(a).
E. Perceptual Study Setup
The perceptual study's 's hardware setup included PC workstation, RME 12mic + RME Digiface AVB, B&K Type 4101 in-ear microphones, Sennheiser HD 800S headphones, and Stream Deck. The study was conducted inside an 8' x 12' Whisper Room, with the monitor inside and the computer placed outside to ensure a controlled, noise-free environment. Custom Matlab and Max MSP patches were developed for use with in-ear mies to create a headphone equalization profile and recreate the recorded signal as accurately as possible. Participants were presented with number of randomly chosen 4-second-loo clips and had 10/10/5 minutes to complete the ABX/MUSHRA/AL sections of the evaluation using Stream Deck and mouse for response input/selection. The setup is shown in Figure 8(b).
F. Pre-training Dataset
We set up 3Dio Omni binaural heads with human- shaped ears non-anechoic recording room. For data collection, operators walked around the room with a handheld loudspeaker, playing speech signals from the English multi-speaker VCTK corpus (Yamagishi al., 2019). Our setup used 135 binaural heads and involved 33 loudspeaker operators. We used the OptiTrack system track the 3D position and orientation of both the loudspeakers and the stationary binaural heads. We collected over 7, 700 hours of binaural audio data, encompassing 97 speaker identities from the VCTK dataset across an area of 4.6m horizontally and 2.4m vertically. The audio was sampled at 48 kHz, with tracking data recorded at 240 frames per second. The setup shown in Figure 9.
G. Comparison with Other Flow Matching-Based Audio Models
PeriodWave (Lee et al., 2024b) designs multi periio flow matching model for high-fidelity waveform generation. FlowDec (Liu et al., 2024) introduces conditional flow matching-based audio codec to noticeably reduce the postfiler DNN evaluations from 60 to 6. RFWave (Welker et al., 2025) proposes multi-band rectified flow approach to reconstruct high-fidelity audio waveforms. These works are all related to flow matching models and show the effectiveness in generating high-quality waveform signals. To reduce the number of sampling steps, Period Wave-Turbo (Lee et al., 2024a) finetunes the CFM models with adversarial feedback. Matcha-TTS (Mehta al., 2024) employs 1D U-Net model with 1D ResNet
14

---

BinauralFlow: A Causal and Streamable Approach for Sigh-Quality Binaural Speech Synthesis with Flow Matching Models
Figure 9. The large-scale scale binaural data capture system with artificial binaural heads.
layers and Transformer Encoder layers. Neither the ResNet layers nor the Transformer Encoder layers are causal, which means that Matcha-TTS does achieve time causality or support streaming inference. In contrast, our model is fully causal and supports streaming inference. Cosy Voice 2 (Du et al., 2024) introduces chunk-aware causal flow matching model that uses causal convolution layers and attention masks to enable causality. However, the Cosy Voice 2 model does not include feature buffers for each causal convolution layer, which may result in audio interruptions and discontinuities during streaming inference in real-world scenarios.
H. Impact of Different Numerical Solvers
Besides the Midpoint solver, test the Fuler and Heun solvers. The Euler solver is first-order solver and Midpoint and Heun solvers second-order. We set the number of function evaluations (NFE) to 6 and present the results in Table 5, Although the Euler solver yields lower error values than the Midpoint solver, it fails to generate realistic background noise. Setting NFE to 6 is insufficient for the Heun solver, which requires 30 steps to achieve comparable error values. In conclusion, the Midpoint solver provides the best trade-off between error values, qualitative results, and inference efficiency.
Table 5. Impact of different numerical solvers. We evaluate our model with various solvers, include hoth first-order and second-order solvers analyze their influence the generation quality.
Solver Type
NFE
Audio Quality
L2 |
Mag J
Phase |
Euler
Medium
0.90
0.0066
1.24
Midpoint
High
1.00
0.0071
1.33
Heun
6
Low
16.86
0.0499
1.44
Heun
30
Medium
1.27
0.0087
1.36
I. Sway Sampling Schedule
Chen et al. (2024) introduce new timestep scheduler called Sway Sampling to improve inference quality and efficiency. We use Sway Sampling with different coefficients ranging from -1 to I to systematically evaluate its impact our model. The results are shown in Table 6. Changing the coefficients does not lead to significant changes in the quantitative results. However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes. Specifically, background noise becomes more realistic when the coefficient is increased. These results support the rationale behind our early skip strategy.
15

---

BinauralFlow: A Causal and Streamable Approach for High-Qualli Binaural Speech Synthesis with Flow Matching Models
Table Impact of Sway Sampling with different coefficients.
Coefficient
L2
Mag |
Phase 4
-1.0
1.06
0.0070
1.29
-0.8
1.10
0.0070
1.29
-0.4
1.00
0.0069
1.29
1.02
0.0069
1.29
0.4
1.03
0.0070
1.31
0.8
1.04
0.0071
1.32
1.0
1.02
0.0072
1.33
J. Results Public Dataset
In the main paper, we compare our BinauralFlow model with existing baselines on our own dataset. To further verify the effectiveness of our approach, we test our model on public dataset released by Richard et (2021). We report the results Table As shown in the table, our model surpasses the state-of-the-art BinauralGrad most of the metries and performs on par with it in the Wave and Phase metrics.
Table 7. Quantitative comparison with existing baselines on the public dataset. Wave is on the scale of x x10
Methods
PESQ t
MRSTFT
Wave L2 |
Amplitude L2
Phase L2 $
DSP
1.610
2.750
1.543
0.097
1.596
WaveNet (Van Den Oord et al., 2016)
2.305
1.915
0.179
0.037
0.968
WarpNet (Richard al., 2021)
2.360
1.774
0.157
0.038
0,838
BinauralGrad (Leng et al., 2022)
2.759
1,278
0.128
0.030
0.837
SGMSE (Richter et al., 2023)
2.256
1.352
0,230
0.033
0,983
BinauralFlow (Ours)
2.806
1.252
0.192
0.030
0.918
K. More Qualitative Results
We display more rendered waveforms in Figures 10 and the audios predicted by different methods between them. Our BinauralFlow model correctly predicts the time delay and audio amplitude.
The first row the mono audio, the last row is the recorded audio,
16

---

BinauralFlow: A Causal and Streamable Approach for High- High-Quality Binaural Speech Synthesis with Flow Matching Models
Left Channel
Right Channel
Mono Audio 0.0
pw tmmmlmm
-0.1
0.1
SoundSpaces
0.0
-0.1
0.1
m mm//mmm Saama/Msaaaa
BinauralGrad
-0.1
0.1
SGMSE
0.0
0.1
www.hMerrr www.heeree-harrrmmmmee hww. Mww.eurrrolleeee
Ours
0.0
4pm
0.1
GT Audio 0.0
Ia WM M Hater
mm Mmn IM Jamm//MMmmm Jaknn//Mtmml
0.1
1000
2000
1000
1500
2000
2500
Figure 10. Qualitative comparison between different baselines. We display waveforms of rendered spatial audio.
Left Channel
Right Chanee
0.1
Mono Audio
-0.1
0.1
SoundSpaces
-0.1
0.1
BinauralGrra
0.0
0.1
SGMSE
0.0
0.1
0.1
Ours
0.0
-0.1
0.1
GT Audio
0.0
0.1
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Figure 11. Qualitative comparison between different baselines. We display waveforms of rendered spatial audio.
17

---

BinauralFlow: A Causal and Streamable Approach for Binaural Speech Synthesis with Flow Matching Models
Left Channel
Right Channel
Mono Audio
0.0
-0.1
SoundSpaces
0.0
0.1
BinauralGrrad
-0.1
0.1
SGMSE
Apo App wpw www. Jlsnaaa
-0.1
0.1
Ours
-0.1
0.1
GT Audio
1000
3000
1000
2000
5000
Figure 12. Qualitative comparison between different basclines. We display waveforms of rendered spatial audio.
18


Reviews:
[{"review_id": "vmRM3U3nqQ", "reviewer": "Reviewer_2E3w", "summary": "They sought to address the binaural speech synthesis task by using mono-channel audio to generate binaural speech. To support streaming and produce audio aligned with a given pose, they employed a flow-matching-based generative model with a causal structure. In the process, they introduced streaming STFT/ISTFT and a buffer bank to enable seamless streaming. Furthermore, to enhance per-chunk generation speed in the flow-matching-based approach, they adopted a solver designed to improve sampling speed and integrated a noise-skip strategy.\n\n## update after rebuttal\nI have reviewed the authors' response. They provided sufficient experimental results and explanations addressing my previous concerns. Therefore, I maintain my positive assessment of this submission.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work. We will address your concerns and include your suggestions in the revision.\n\n**Dataset** (Questions For Authors)\n\nRegarding open-sourcing the dataset, we fully understand the importance of reproducibility and transparency in research. However, due to privacy constraints and participant confidentiality, we are unable to publicly release the full dataset. To ensure the reproducibility of our work, we will release **all implementation code, training scripts, pretrained model weights, and a test subset** that has been carefully curated to exclude any personal information.\n\n**Different solvers** (Questions For Authors)\n\nBesides the Midpoint solver, we test the Euler and Heun solvers. The Euler solver is a first-order solver and Midpoint and Heun solvers are second-order. We set the number of function evaluations (NFE) to 6 and present the results below. Although the Euler solver yields lower error values than the Midpoint solver, it fails to generate realistic background noise. Setting NFE to 6 is insufficient for the Heun solver, which requires 30 steps to achieve comparable error values. In conclusion, the Midpoint solver provides the best trade-off between error values, qualitative results, and inference efficiency.\n| Solver Type | NFE |Quality | L2 $\\downarrow$ | Mag $\\downarrow$ | Phase $\\downarrow$ \n|:-|:-:|:-:|:-:|:-:|:-:|\n| Euler | 6 | Medium | 0.90 | 0.0066 | 1.24 |\n| Midpoint | 6 |  High | 1.00 | 0.0071 | 1.33 |\n| Heun | 6 | Low | 16.86 | 0.0499 | 1.44 |\n| Heun | 30 | Medium | 1.27 | 0.0087 | 1.36 |\n\n**STFT-based complex spectrograms** (Questions For Authors)\n\nA mel spectrogram is derived by mapping the STFT-based complex spectrogram to the mel scale. In this process, only the magnitude is retained, while the phase is discarded. Spatial audio rendering relies on precise phase information to capture interaural time differences (ITD) and interaural phase differences (IPD). These phase cues are essential for accurately conveying spatial positioning in binaural audio, ensuring a realistic 3D auditory experience. Therefore, we use STFT-based complex spectrograms, which retain both magnitude and phase information.\n\n**Latency** (Questions For Authors)\n\nWe test the inference latency of our streaming model using a single 4090 GPU. The audio sampling rate is 48 kHz and the audio length is 0.683 seconds. We vary the NFE from 1 to 10 and report the corresponding inference times below. We also report the real-time factor (RTF), which is calculated by dividing the processing time by the actual time. If RTF is less than 1, the system runs faster than real-time. With NFE set to 6, the real-time factor is 0.239. Reducing NFE to 1 improves inference speed at the cost of some performance, yielding an RTF of 0.040. These results demonstrate our model's potential for real-time streaming generation.\n| NFE | Inference Time (sec) | Real-Time Factor|\n|:-:|:-:|:-:|\n| 1 | 0.027 | 0.040 |\n| 2 | 0.055 | 0.081 |\n| 4 | 0.109 | 0.160 |\n| 6 | 0.163 | 0.239 |\n| 8 | 0.217 | 0.318 |\n| 10 | 0.271 | 0.397 |\n\n\n**Streaming vs non-streaming models** (Questions For Authors)\n\nFor a sequence of audio chunks, the streaming model buffers intermediate features to enable seamless streaming inference, while non-streaming model processes each audio chunk independently without buffering. We visualize the spectrograms generated by both the streaming and non-streaming models in Figure 7 of the main paper, where the non-streaming models generate noticeable artifacts between audio chunks. Below, we present a quantitative comparison between the two. The streaming model achieves better audio quality than the non-streaming version, producing smoother and more continuous audio generation.\n\n| Methods | L2 $\\downarrow$ | Mag $\\downarrow$ | Phase $\\downarrow$ |\n|:-|:-:|:-:|:-:|\n|Streaming | $\\mathbf{1.00}$ | $\\mathbf{0.0071}$ | $\\mathbf{1.33}$|\n|Non-streaming | 13.25 | 0.0398 | 1.34 |", "claims_and_evidence": "At least, there do not appear to be any issues with the authors’ claims.", "methods_and_evaluation_criteria": "There also seem to be no issues with the methodology or the evaluation.", "theoretical_claims": "It appears that, rather than putting forth a separate theoretical argument, they have primarily adopted existing theoretical backgrounds (e.g., flow matching, midpoint solver), which seems reasonable.", "experimental_designs_or_analyses": "The experimental design and analysis also appear to be reasonable.", "relation_to_broader_scientific_literature": "They appear to have achieved similar or even improved quality compared to previous studies, while also providing streaming support.", "essential_references_not_discussed": "I do not see anything else noteworthy in that regard.", "other_strengths_and_weaknesses": "**[S1]** The figures and explanations are well-presented, making it easy for individuals who are not familiar with binaural speech synthesis to understand.\n\n**[S2]** It is also commendable that the authors collected new data for verification, and that they provide results on publicly available datasets (as mentioned in the Appendix).\n\n**[S3]** Moreover, enabling streaming capability further enhances the model’s practical utility.\n\nI will outline my questions below.", "other_comments_or_suggestions": "I will outline my questions below.", "questions_for_authors": "**[Q1]** I am curious whether you plan to make the collected dataset publicly available.\n\n**[Q2]** Could you provide any results comparing the midpoint solver with other solvers?\n\n**[Q3]** Since I am not very familiar with this domain, I wonder why you use STFT-based complex spectrograms rather than mel spectrograms, which are more common in typical speech synthesis (TTS).\n\n**[Q4]** Finally, I would like to know more about the latency of the streaming model and how it compares in terms of performance with the non-streaming version.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MmWdHXqIpB", "reviewer": "Reviewer_jaNd", "summary": "This paper proposes a streaming binaural speech synthesis method using a causal architecture design and flow matching models. It introduces a flow matching model to generate binaural speech from a single-channel input. Additionally, it adopts a causal architecture to predict the next frames based on past information for streaming inference. To efficiently estimate the vector fields, buffers for the features at each sampling step are stacked, and only the current frame is computed using them. The results show better performance than other baselines.\n\n## Update after rebuttal\n\nI remain positive about the paper and will keep my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive feedback on our work. We will address your questions in the responses below.\n\n**Related work** (Claims And Evidence)\n\nWe thank the reviewer for pointing out these related works. PeriodWave designs a multi-period flow matching model for high-fidelity waveform generation. FlowDec introduces a conditional flow matching-based audio codec to noticeably reduce the postfiler DNN evaluations from 60 to 6. RFWave proposes a multi-band rectified flow approach to reconstruct high-fidelity audio waveforms. These works are all related to flow matching models and show the effectiveness in generating high-quality waveform signals. We will include our discussion in the revised paper.\n\n**Real-time factor** (Claims And Evidence, Methods And Evaluation Criteria)\n\nWe calculate the real-time factor of our model for different numbers of function evaluations on a single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 0.683 seconds. As shown in the table, when NFE is set to 6, the real-time factor is 0.239. If we sacrifice some performance for faster inference, setting NFE to 1 results in an RTF of 0.04. Our model demonstrates potential for real-time streaming generation.\n| NFE | Inference Time (sec) | Real-Time Factor|\n|:-:|:-:|:-:|\n| 1 | 0.027 | 0.040 |\n| 2 | 0.055 | 0.081 |\n| 4 | 0.109 | 0.160 |\n| 6 | 0.163 | 0.239 |\n| 8 | 0.217 | 0.318 |\n| 10 | 0.271 | 0.397 |\n\n**Discussion on Matcha-TTS and CosyVoice 2** (Claims And Evidence)\n\nWe carefully reviewed the papers and code of these two works. Matcha-TTS employs a 1D U-Net model with 1D ResNet layers and Transformer Encoder layers. Neither the ResNet layers nor the Transformer Encoder layers are causal, which means that Matcha-TTS does not achieve time causality or support streaming inference. In contrast, our model is fully causal and supports streaming inference.\n\nCosyVoice 2 introduces a chunk-aware causal flow matching model that uses causal convolution layers and attention masks to enable causality. However, the CosyVoice 2 model does not include feature buffers for each causal convolution layer, which may result in audio interruptions and discontinuities during streaming inference in real-world scenarios.\n\nWe will include our discussion of Matcha-TTS and CosyVoice 2 in our revision.\n\n**Experiment of parallel model** (Claims And Evidence, Methods And Evaluation Criteria)\n\nWe compare the model's performance with and without buffers to examine the effectiveness of our causal model design. The results show that BinauralFlow with buffers achieves higher quality than the model without buffers. Additionally, in Figure 7 in the main paper, we show that excluding buffers causes noticeable artifacts in the generated spectrograms.\n| Methods | L2 $\\downarrow$ | Mag $\\downarrow$ | Phase $\\downarrow$ |\n|:-|:-:|:-:|:-:|\n|BinauralFlow w/ buffer | $\\mathbf{1.00}$ | $\\mathbf{0.0071}$ | $\\mathbf{1.33}$|\n|BinauralFlow w/o buffer | 13.25 | 0.0398 | 1.34 |\n\n**Sway Sampling** (Claims And Evidence)\n\nAs suggested by the reviewer, we use Sway Sampling with different coefficients ranging from -1 to 1 to systematically evaluate our model. The results are shown in the table below. Changing the coefficients does not lead to significant changes in the quantitative results. However, we observe that setting coefficients greater than 0, which shifts the time steps to the second half, results in better qualitative outcomes. Specifically, background noise becomes more realistic when the coefficient is increased. These results support the rationale behind our early skip strategy.\n\n| Coefficients | L2 $\\downarrow$ | Mag $\\downarrow$ | Phase $\\downarrow$ |\n|:-|:-:|:-:|:-:|\n| -1.0 | 1.06 | 0.0070 | 1.29 |\n| -0.8 | 1.10 | 0.0070 | 1.29 | \n| -0.4 | 1.00 | 0.0069 | 1.29 |\n| 0 | 1.02 | 0.0069 | 1.29 |\n| 0.4 | 1.03 | 9.0070 | 1.31 |\n| 0.8 | 1.04 | 0.0071 | 1.32 |\n| 1.0 | 1.02 | 0.0072 | 1.33 |\n\n**Sound dataset** (Experimental Designs Or Analyses)\n\nThank you for the suggestion. We plan to conduct this experiment in our future work.\n\n**Demo** (Supplementary Material)\n\nAs recommended by the reviewer, we have created a demo page and will release it following the acceptance of our paper.\n\n**GAN-based models** (Questions For Authors)\n\nUtilizing GAN-based models, such as PeriodWave-Turbo, to enhance inference efficiency is a promising direction. We plan to discuss this possibility in our paper and explore it further in future work.\n\n**Reshape and linear projection** (Questions For Authors)\n\nWe tried using waveform directly without STFT/iSTFT but it did not lead to superior performance. We are still interested in waveform-based approaches. We will explore the reshape method and linear projection proposed in WaveNeXt and discuss them in our paper.", "claims_and_evidence": "[Flow Matching for Binaural Audio Generation]\n\nBased on previous work [1], [2], [3], flow matching models have already been verified to effectively generate high-quality waveform signals. The related works on flow matching-based waveform generation are not discussed in the current manuscript.\n\n[1] https://openreview.net/forum?id=tQ1PmLfPBL\n\n[2] https://openreview.net/forum?id=gRmWtOnTLK\n\n[3] https://openreview.net/forum?id=uxDFlPGRLX\n\n[Future Frame Generation based on the past information]\n\nIn terms of generative tasks, this paper adopts future frame generation using flow matching based solely on past information. This approach has potential for streaming generation in real-time applications. However, the current paper does not describe the real-time capabilities; it only discusses streaming generation in terms of a smaller NFE and its causal design. Please evaluate the real-time factor for streaming generation, as it is important to demonstrate real-time streaming ability if the paper claims to support streaming generation.  \n\n[Causal Model Design with Buffer]\n\nI like the proposed causal design for streaming generation using buffers at each sampling step. However, this structure was already proposed in CosyVoice 2, which employs a chunk-aware causal flow matching model for streaming synthesis. Furthermore, the model structure is identical to that of Matcha-TTS based UNet architectures. However, this paper does not refer Matcha-TTS. Please discuss the differences between the proposed models and those in Matcha-TTS and CosyVoice 2. CosyVoice 2 was released at Dec. 2024 so I just think that this work is a concurrent work. Feel free to add the discussion about this.\n\n[4] Mehta, Shivam, et al. \"Matcha-TTS: A fast TTS architecture with conditional flow matching.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n[5] Du, Zhihao, et al. \"Cosyvoice 2: Scalable streaming speech synthesis with large language models.\" arXiv preprint arXiv:2412.10117 (2024).\n\n[Comparison with Parallel Model]\n\nIt would be better if the ablation study with a parallel model is added.\n\n[Early Skip]\n\nThis paper only empirically claims the early skip strategy could improve the performance. Also, the ratio of 0.5 was chosen heuristically. I recommend using the Sway Sampling of F5-TTS using different coefficient.  \n\n[6] Chen, Yushen, et al. \"F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching.\" arXiv preprint arXiv:2410.06885 (2024).", "methods_and_evaluation_criteria": "Evaluation metrics are limited in supporting the streaming ability. Please added more evaluation metrics regarding the real-time factor, and compare the performance with or without the buffer to support the efficient causal model design.", "theoretical_claims": ".", "experimental_designs_or_analyses": "The evaluation might be conducted on the speech dataset. It would be beneficial to include results from a sound dataset as well.", "relation_to_broader_scientific_literature": ".", "essential_references_not_discussed": "[1] https://openreview.net/forum?id=tQ1PmLfPBL\n\n[2] https://openreview.net/forum?id=gRmWtOnTLK\n\n[3] https://openreview.net/forum?id=uxDFlPGRLX\n\n[4] Mehta, Shivam, et al. \"Matcha-TTS: A fast TTS architecture with conditional flow matching.\" ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024.\n\n[5] Du, Zhihao, et al. \"Cosyvoice 2: Scalable streaming speech synthesis with large language models.\" arXiv preprint arXiv:2412.10117 (2024).", "other_strengths_and_weaknesses": ".", "other_comments_or_suggestions": "The concept of this paper is good to me. However, the details for real-time streaming generation and the survey of the related works are limited. Please discuss the related works more and evaluate the real-time factor to demonstrate the concept of this paper. \n\nIn the future, GPUs will become much faster so iterative sampling methods for waveform generation will be used more practically. However, I hope that it is not the overstated claims for the streaming ability in the current state. Please discuss the real-time ability.", "questions_for_authors": "[Q1: GAN-based models]\nHave you compared the model with GAN-based models? Although CFM-based models can generate high-quality waveforms, GAN-based models are still effective for waveform generation. Specifically, PeriodWave-Turbo [7] fine-tuned the CFM models with adversarial feedback to improve performance and reduce the number of sampling steps.\n\nFor real-time applications, this model still requires a midpoint method with six NFEs, resulting in higher latency.\n\n[Q2: Reshape and Linear Projection methods instead of  STFT/iSTFT]\n\nHave you tried using different input features instead of STFT components? I recommend using the waveform directly with a reshape method and linear projection based on WaveNeXt [8]. This approach does not require a reflected padding before STFT.\n\n[7] Lee, Sang-Hoon, Ha-Yeong Choi, and Seong-Whan Lee. \"Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization.\" arXiv preprint arXiv:2408.08019 (2024).\n\n[8] Okamoto, Takuma, et al. \"WaveNeXt: ConvNeXt-based fast neural vocoder without iSTFT layer.\" 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "."}}, {"review_id": "BdV8JRIggC", "reviewer": "Reviewer_j2YT", "summary": "This paper explores the streaming generation of high-quality binaural audio from monaural audio, considering the spatial positions of both the speaker and listener. Specifically, the task is approached as a generative problem, utilizing a flow matching model to generate stochastic binaural details absent in the monaural input. The paper also proposes a continuous inference pipeline to support streaming rendering. Both objective and subjective metrics demonstrate the superiority of the proposed approach.\n\n## Update after rebuttal\n\nThank you for your rebuttal response. I appreciate the added experimental results regarding inference speed. I suggest incorporating these into the main paper. Though, I still have reservations about the method motivation and the technical distinction between the proposed method and the previous works. Therefore, I will maintain my original score.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your constructive comments. We will address your concerns below and will revise our paper following your suggestions.\n\n**Performance on the public dataset** (Claims And Evidence)\n\nWe agree that the proposed method performs comparably to the baseline (BinauralGrad) on some metrics, but we respectfully clarify that our approach outperforms the baseline on **key perceptual metrics** (PESQ and MRSTFT). These percetual metrics are often prioritized in audio synthesis tasks as they correlate better with human perception of quality and intelligibility, whereas minor waveform-level differences (L2) may not impact perceived audio quality.\n\n**Implementation details of baselines** (Claims And Evidence)\n\nFor all baselines, we trained the models from scratch on our new dataset and tested them on the same dataset. We did not perform pretraining or cross-dataset evaluation. We will include additional implementation details in our revision.\n\n**Model size and speed** (Claims And Evidence)\n\nWe thank the reviewer for pointing out this issue. We present the model size and inference speed for all baselines below. We test the inference speed on a single 4090 GPU. The audio sampling rate is 48 kHz, and the audio length is 683 ms. As shown in the table, our model achieves the fastest inference speed among generative models. Our model acheives a more favorable trade-off between performance (Table 1) and inference speed compared to the baseline approaches.\n\n|Methods|Type|NFE|Speed (ms)|Model Size (MB)|\n|:-|:-:|:-:|:-:|:-:|\n|SoundSapces 2.0|-|1|-|-|\n|2.5D Visual Sound|R|1|1.1|82.0|\n|WaveNet|R|1|21.0|32.7|\n|WarpNet|R|1|21.9|32.8|\n|BinauralGrad|G|6|221.1|52.9|\n|SGMSE|G|30|770.2|273.6|\n|BinauralFlow (Ours)|G|6|163.0|314.5|$\\mathbf{1.00}$|$\\mathbf{0.0071}$|$\\mathbf{1.33}$|\n\n**Method motivation** (Methods And Evaluation Criteria, Other Strengths And Weaknesses)\n\nReal-time, high-quality spatial audio is essential for immersive applications such as gaming, VR/AR, and cinematic experiences. These applications motivate the design of a high-quality, efficient, and streaming binaural audio generation framework.\n\nFor **quality**, we frame the binaural audio rendering task as a generative problem and introduce a high-fidelity generative model. To improve inference **efficiency**, we choose flow matching models over diffusion models because they render high-fidelity spatial audio with fewer inference steps. We further reduce inference steps by using the midpoint solver and an early skip strategy. To satisfy the **streaming** requirement of real-world applications, we design our causal U-Net architecture and continuous inference pipeline that processes audio input in chunks. Our method is designed to meet real-world needs, and our model successfully addresses these challenges.\n\n**Use of SGMSE baseline** (Experimental Designs Or Analyses)\n\nAlthough SGMSE is not intended for binaural audio rendering, we included it as a baseline due to its U-Net-based diffusion model. While BinauralGrad also uses a diffusion model, it employs a WaveNet architecture. We used SGMSE to examine the impact of different architectures on the results.\n\n**Omission of SGMSE results in Table 4** (Experimental Designs Or Analyses)\n\nSince SGMSE was not evaluated on the public dataset, it was not included in Table 4. To expand our experiment, we test SGMSE on the public dataset and report its performance below. Our model outperforms SGMSE with noticeable margins. We will include these results in the revision.\n|Methods|PESQ$\\uparrow$|MRSTFT$\\downarrow$|Wave L2$\\downarrow$|Amplitude L2$\\downarrow$|Phase L2$\\downarrow$|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|SGMSE|2.256|1.352|0.230|0.033|0.983|\n|BinauralFlow (Ours)|2.806|1.252|0.192|0.030|0.918|\n\n**L2 error scale** (Experimental Designs Or Analyses)\n\nThe difference in scale is due to the audio volume in the two datasets not being the same. Since the audio volume in our collected dataset is lower than in the public dataset, we used a different scaling factor during testing.\n\n**Difference between BinauralFlow and simplified flow** (Questions For Authors)\n\nThe key difference between our approach and the simplified flow lies in the flow function and vector field. Our flow function is defined as $\\phi_t(\\mathbf{z}) = t\\mathbf{y} + (1-t)\\mathbf{x} + (1-t)\\sigma \\mathbf{\\epsilon}$, where $\\mathbf{x}$ and $\\mathbf{y}$ are mono and binaural audio, respectively, $\\mathbf{\\epsilon}$ is Gaussian noise, $t$ is the time step, and $\\sigma = 0.5$ in our experiments. The simplified flow uses $\\phi_t(\\mathbf{z}) = t\\mathbf{y} + (1-t)\\mathbf{x} + \\sigma \\mathbf{\\epsilon}$ with $\\sigma$ near zero (e.g., $1e{-}4$), reducing to a deterministic interpolation $\\phi_t(\\mathbf{z}) = t\\mathbf{y} + (1-t)\\mathbf{x}$. Consequently, our vector field $\\mathbf{y}-\\mathbf{x}-\\sigma\\mathbf{\\epsilon}$ introduces stochasticity, enabling variability in generative tasks, unlike the deterministic $\\mathbf{y}-\\mathbf{x}$ in the simplified flow.", "claims_and_evidence": "The paper claims to achieve high-quality and realistic binaural audio rendering with faster inference speed. The evidence provided includes performance comparisons on an in-house test set. \n\nHowever, the evidence is not sufficiently convincing for several reasons. First, the evaluation results on a public dataset (Appendix F) show that the proposed approach performs on par with the baseline, except for the Wave L2 metric, which is significantly worse than the baseline. This suggests that the overall performance improvement may be limited. Second, necessary details about the baseline model are missing. For example, it is unclear whether the baseline model was trained on the same dataset or tested in an out-of-domain scenario. Additionally, the model size is not reported, which is essential for evaluating inference speed, as model size directly impacts computational efficiency. These details are crucial for a fair comparison and to validate the claims of improved performance and speed.", "methods_and_evaluation_criteria": "The enhancements in effectiveness and efficiency are indeed valuable for the binaural audio rendering task and practical applications, though a more thorough exploration of the underlying motivations of the proposed methods would help clarify the rationale and ensure the contributions are fully understood.", "theoretical_claims": "This work primarily focuses on practical application rather than theoretical development. I specifically review the formulas related to flow matching, and they seem to be correctly formulated.", "experimental_designs_or_analyses": "Some questions about the experimental design are mentioned in the “Claims and Evidence” section.  Additionally, I would like to understand the rationale behind the choice of the SGMSE baseline, since its original paper may not explicitly address the binaural audio task. I am also confused by the omission of SGMSE results in Table 4. Lastly, I suggest clarifying the reasons for the inconsistent scales used in the L2 error reporting (1e-5 in Table 1 vs. 1e-3 in Table 4).", "relation_to_broader_scientific_literature": "The key contributions of this paper build on prior work by continuing the paradigm of framing this task as a generative problem. The authors focus on adapting the model architechture and some implementation details to enhance both performance and efficiency.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper demonstrates notable improvements in subjective testing, which underscores the effectiveness of the proposed approach. Overall, however, the technical contributions appear relatively incremental. The paper could benefit from a more thorough discussion of the motivations behind the proposed improvements. To some extent, the work reads more as a technical report, which, while valuable, may limit its broader impact.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. The discussion on the differences between this work and simplified flow matching in Section 3.2 is somewhat perplexing, particularly regarding the inclusion of task-related conditions (such as mono audio). The current text appears to emphasize experimental results, such as the impact of noise scale variations on training stability, rather than clearly articulating the distinctions between the two methods. Could the authors further clarify these distinctions to provide a clearer understanding of their respective approaches and implementations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a model for generating high quality binaural speech with flow matching models allowing for streaming applications by applying a series of tricks to improve rendering continuity and speed. Even though some of the reviewers have pointed out several similarities between this work and other works, there are still specific ideas proposed here that seem to be novel (although of limited interest). This is definitely a weak accept as the work is solid but of limited interest.
Decision: Accept (poster)