aryanvibhosale/vibe

VIBE: Video Instruction-aligned Background Music Generation

🎉 Accepted to Findings of EMNLP 2026

Description

VIBE generates background music for a video that follows an explicit text instruction, not just "music that fits this video", but music that fits and does what the user asked. It extends a tokenizer-free, continuous-latent music model with reinforcement learning against a multimodal LLM judge and rule-based verifiable rewards, so the generated music aligns with both the video and the instruction.

Rather than converting audio to discrete tokens, VIBE models music in a continuous latent space: a multimodal semantic LM produces planning latents, a RITE (Residual Integration Transformer Encoder) stack refines them, and a local diffusion transformer, the LocDiT, denoises under conditional flow matching before a 48 kHz stereo VAE decodes to audio. Video conditioning enters through a CLIP-based encoder, and semantic routing merges the video and instruction signals into a single conditioning stream.

Built on VoxCPM (Apache-2.0), adapted from speech to music.

• Instruction-Aligned Music Generation: conditions on a video and a free-form text instruction, so tempo, key, mood, and instrumentation can be requested explicitly rather than inferred from the video alone.
• Reinforcement Learning from a Multimodal Judge: GRPO against Qwen2.5-Omni-7B, which watches the video and listens to the generated music, scoring musicality, text-music alignment, and video-music alignment. Scores come from constrained digit-logit decoding, so there are no JSON parse failures.
• Hard Verifiable Rewards: rule-based tempo/BPM and musical-key agreement, measured on the generated audio with librosa/essentia and scored against targets parsed from the instruction. Deterministic, CPU-only, no learned parameters, and immune to reward hacking.
• Continuous Latents, No Audio Tokenizer: 48 kHz stereo output through a music VAE, avoiding the quality ceiling and codebook artifacts of discrete tokenization.

Model Architecture

_{VIBE Architecture. Video frames, text prompt, and previously generated patch embeddings are passed to the Multimodal Semantic LM. Layer-wise hidden states are linearly combined via learnable per-DiT-layer coefficients to form Conditioning Connectors, which are routed to every LocDiT layer alongside residual-integrated embeddings to generate each music patch.}

Semantic LM	MiniCPM4-0.5B architecture: 24 layers, hidden 1024, 16 heads, vocab 73 448
RITE	8 layers, hidden 1024
LocDiT	8 layers, conditional flow matching, Euler solver
Semantic routing	8 LocDiT blocks × learned softmax over 24 LM layers
Latents	patch size 4, feature dim 64, FSQ latent dim 256
Video encoder	CLIP ViT-B/32, 8 sampled frames, projection dim 512 (frozen, weights embedded)
Audio decoder	SongBloom / Stable Audio VAE (frozen, external)
Output	48 kHz stereo
Parameters	953.5 M across 873 tensors (3.51 GB)
Precision	bf16 at inference; generator stored fp32, CLIP encoder bf16

VIBE is a custom architecture. It is not loadable with transformers.AutoModel; use the inference scripts in the VIBE repository.

Training

_{Overview of our training curriculum.}

Training data: JamendoMaxCaps (text-to-music pretraining), MusicBench (text-to-music SFT), CMI-Pref (captions for text-to-music preference optimisation), and HarmonySet (video-to-music). Evaluation is on ReelBench.

Provenance

This is the Stage-5 RL policy. The RL LoRA adapter (r=8, α=16, on q_proj and v_proj of both the LM and the LocDiT) has already been folded into the base weights using W += (α/r) · B·A, so it loads as a plain model with no LoRA machinery.

⚠️ Do not pass --lora_weights_path against this checkpoint. The adapter is already merged; supplying it again would apply it a second time.

Usage

This repository holds weights only, no code. Clone the code repository, then point --ckpt_dir at a download of this model.

git clone https://github.com/aryanvibhosale/vibe.git && cd vibe
conda env create -f environment.yml && conda activate vibe

Requires transformers>=5.0.0. The video encoder relies on CLIPModel.get_image_features() returning a pooled-output object, which is v5 behaviour. On transformers 4.x, video-to-music fails with AttributeError: 'Tensor' object has no attribute 'pooler_output'.

If environment creation fails on the transformers pin, install it separately: pip install "transformers>=5.0.0".

Download the weights and the external music VAE:

hf download aryanvibhosale/vibe --local-dir ./vibe_ckpt

hf download rsxdalv/SongBloom \
    --include "autoencoder_music_dsp1920.ckpt" "stable_audio_1920_vae.json" \
    --local-dir ./music_vae_cache

Video-to-music. Video plus instruction in, music out:

python scripts/infer_v2m.py \
    --ckpt_dir      ./vibe_ckpt \
    --audiovae_path ./music_vae_cache \
    --text          "An ambient electronic track at 125 BPM in B Minor." \
    --video_path    /path/to/input_video.mp4 \
    --output        v2m_out.wav

Text-to-music. Instruction only, no video. Use scripts/infer_ttm.py with the same --ckpt_dir and --audiovae_path.

Tune quality with --cfg_value (default 2.0) and --inference_timesteps (default 10 for video-to-music, 20 for text-to-music).

Only the first 10 seconds of --video_path are used; 8 frames are sampled uniformly from that window. Longer clips are truncated.

Files in this repository

File
model.safetensors	3.51 GB, 873 tensors, 953.5 M parameters
config.json	model configuration
tokenizer.json, tokenizer_config.json, special_tokens_map.json	LlamaTokenizerFast
static/	model card images, not used by the model

Required external components

Component	Where	Passed as
SongBloom / Stable Audio music VAE	rsxdalv/SongBloom (Apache-2.0)	--audiovae_path
openai/clip-vit-base-patch32 preprocessor config	Hugging Face, fetched at init (a few kB)	n/a

--audiovae_path is a directory holding autoencoder_music_dsp1920.ckpt and stable_audio_1920_vae.json. It is deliberately not bundled here: the audio VAE is never read from the checkpoint directory, and its weights carry their own licence.

The CLIP weights are embedded in model.safetensors. Only the small image preprocessor config is fetched from the Hub at init, so a populated HF_HOME cache is enough for offline use. You do not need to download MiniCPM4-0.5B: the semantic LM weights are in model.safetensors and the tokenizer ships here.

Limitations

• Instruction adherence is strongest for tempo. Automatic musical-key detection is unreliable across the board, including on real reference audio, so key agreement should not be read as a headline metric.
• RL trades distributional metrics for adherence and diversity. The reinforcement-learned policy is not tuned to minimise FAD or maximise ImageBind similarity, and should not be ranked by them or framed as "closest to ground truth".
• Generated audio may be unexpected or contain artifacts, particularly for instructions far from the training distribution. Released for research purposes; we do not recommend production use without further testing.
• Short-form instrumental music. Trained for background music clips, not full arrangements or vocals. Output is instrumental; the en language tag refers to the text instruction, not to the audio.
• Respect the rights in your inputs. Generating music conditioned on video you do not have rights to, or producing music that imitates a specific artist's protected work, may infringe. We recommend clearly marking generated audio as AI-generated.

License

Released under the Apache-2.0 license.

These weights derive from MiniCPM4-0.5B (Apache-2.0) and embed a frozen CLIP ViT-B/32 (MIT) video encoder. The Qwen2.5-Omni judge and CMI-RM (CMI-RewardBench) shaped training as reward signals only and are not distilled into these weights; neither is redistributed here. CMI-RM's published weights are released for non-commercial use; consult the upstream repository for its current terms. Full third-party attribution is in the NOTICE file. The audio VAE is not included; see Required external components.

Acknowledgements

This work builds on open-source research and code. With thanks to: VoxCPM · MiniCPM-4 · CosyVoice · DAC · Qwen2.5-Omni · MuQ / MuQ-MuLan · CMI-RewardBench