TalkingMachines: Real-Time Audio-Driven FaceTime-Style Video via
  Autoregressive Diffusion Models
Authors: name: Chetwin Low, name: Weimin Wang

In this paper, we present TalkingMachines -- an efficient framework that
transforms pretrained video generation models into real-time, audio-driven
character animators. TalkingMachines enables natural conversational experiences
by integrating an audio large language model (LLM) with our video generation
foundation model. Our primary contributions include: (1) We adapt a pretrained
SOTA image-to-video DiT into an audio-driven avatar generation model of 18
billion parameters; (2) We enable infinite video streaming without error
accumulation through asymmetric knowledge distillation from a bidirectional
teacher model into a sparse causal, autoregressive student model; (3) We design
a high-throughput, low-latency inference pipeline incorporating several key
engineering optimizations such as: (a) disaggregation of the DiT and VAE
decoder across separate devices, (b) efficient overlap of inter-device
communication and computation using CUDA streams, (c) elimination of redundant
recomputations to maximize frame-generation throughput. Please see demo videos
here - https://aaxwaz.github.io/TalkingMachines/