BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
  Speech Synthesis with Flow Matching Models
Authors: name: Susan Liang; name: Dejan Markovic; name: Israel D. Gebru; name: Steven Krenn; name: Todd Keebler; name: Jacob Sandakly; name: Frank Yu; name: Samuel Hassel; name: Chenliang Xu; name: Alexander Richard
Published: 2025-05-28 20:59:15

Abstract: Binaural rendering aims to synthesize binaural audio that mimics natural
hearing based on a mono audio and the locations of the speaker and listener.
Although many methods have been proposed to solve this problem, they struggle
with rendering quality and streamable inference. Synthesizing high-quality
binaural audio that is indistinguishable from real-world recordings requires
precise modeling of binaural cues, room reverb, and ambient sounds.
Additionally, real-world applications demand streaming inference. To address
these challenges, we propose a flow matching based streaming binaural speech
synthesis framework called BinauralFlow. We consider binaural rendering to be a
generation problem rather than a regression problem and design a conditional
flow matching model to render high-quality audio. Moreover, we design a causal
U-Net architecture that estimates the current audio frame solely based on past
information to tailor generative models for streaming inference. Finally, we
introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
operations, a buffer bank, a midpoint solver, and an early skip schedule to
improve rendering continuity and speed. Quantitative and qualitative
evaluations demonstrate the superiority of our method over SOTA approaches. A
perceptual study further reveals that our model is nearly indistinguishable
from real-world recordings, with a 42% confusion rate.