SoundReactor: Frame-level Online Video-to-Audio Generation
Authors: name: Koichi Saito, name: Julian Tanke, name: Christian Simon, name: Masato Ishii, name: Kazuki Shimada, name: Zachary Novack, name: Zhi Zhong, name: Akio Hayakawa, name: Takashi Shibuya, name: Yuki Mitsufuji
Organization: Sony

Prevailing Video-to-Audio (V2A) generation models operate offline, assuming
an entire video sequence or chunks of frames are available beforehand. This
critically limits their use in interactive applications such as live content
creation and emerging generative world models. To address this gap, we
introduce the novel task of frame-level online V2A generation, where a model
autoregressively generates audio from video without access to future video
frames. Furthermore, we propose SoundReactor, which, to the best of our
knowledge, is the first simple yet effective framework explicitly tailored for
this task. Our design enforces end-to-end causality and targets low per-frame
latency with audio-visual synchronization. Our model's backbone is a
decoder-only causal transformer over continuous audio latents. For vision
conditioning, it leverages grid (patch) features extracted from the smallest
variant of the DINOv2 vision encoder, which are aggregated into a single token
per frame to maintain end-to-end causality and efficiency. The model is trained
through a diffusion pre-training followed by consistency fine-tuning to
accelerate the diffusion head decoding. On a benchmark of diverse gameplay
videos from AAA titles, our model successfully generates semantically and
temporally aligned, high-quality full-band stereo audio, validated by both
objective and human evaluations. Furthermore, our model achieves low per-frame
waveform-level latency (26.3ms with the head NFE=1, 31.5ms with NFE=4) on
30FPS, 480p videos using a single H100. Demo samples are available at
https://koichi-saito-sony.github.io/soundreactor/.