Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video
Authors: name: Ciara Rowles; name: Varun Jampani; name: Simon Donné; name: Shimon Vainer; name: Julian Parker; name: Zach Evans
Published: 2025-10-24 15:49:54

Abstract: Foley Control is a lightweight approach to video-guided Foley that keeps
pretrained single-modality models frozen and learns only a small
cross-attention bridge between them. We connect V-JEPA2 video embeddings to a
frozen Stable Audio Open DiT text-to-audio (T2A) model by inserting compact
video cross-attention after the model's existing text cross-attention, so
prompts set global semantics while video refines timing and local dynamics. The
frozen backbones retain strong marginals (video; audio given text) and the
bridge learns the audio-video dependency needed for synchronization -- without
retraining the audio prior. To cut memory and stabilize training, we pool video
tokens before conditioning. On curated video-audio benchmarks, Foley Control
delivers competitive temporal and semantic alignment with far fewer trainable
parameters than recent multi-modal systems, while preserving prompt-driven
controllability and production-friendly modularity (swap/upgrade encoders or
the T2A backbone without end-to-end retraining). Although we focus on
Video-to-Foley, the same bridge design can potentially extend to other audio
modalities (e.g., speech).