BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural
  Speech Synthesis with Flow Matching Models
Authors: name: Susan Liang, name: Dejan Markovic, name: Israel D. Gebru, name: Steven Krenn, name: Todd Keebler, name: Jacob Sandakly, name: Frank Yu, name: Samuel Hassel, name: Chenliang Xu, name: Alexander Richard

Binaural rendering aims to synthesize binaural audio that mimics natural
hearing based on a mono audio and the locations of the speaker and listener.
Although many methods have been proposed to solve this problem, they struggle
with rendering quality and streamable inference. Synthesizing high-quality
binaural audio that is indistinguishable from real-world recordings requires
precise modeling of binaural cues, room reverb, and ambient sounds.
Additionally, real-world applications demand streaming inference. To address
these challenges, we propose a flow matching based streaming binaural speech
synthesis framework called BinauralFlow. We consider binaural rendering to be a
generation problem rather than a regression problem and design a conditional
flow matching model to render high-quality audio. Moreover, we design a causal
U-Net architecture that estimates the current audio frame solely based on past
information to tailor generative models for streaming inference. Finally, we
introduce a continuous inference pipeline incorporating streaming STFT/ISTFT
operations, a buffer bank, a midpoint solver, and an early skip schedule to
improve rendering continuity and speed. Quantitative and qualitative
evaluations demonstrate the superiority of our method over SOTA approaches. A
perceptual study further reveals that our model is nearly indistinguishable
from real-world recordings, with a 42% confusion rate.