vibe-embed v1.2
Search music by how it sounds and feels: "rainy Sunday coffee shop", "upbeat but melancholic, no vocals", "something for a long night drive". Songs and descriptions share one 512-number vector space, so you can search a library by text or find songs with the same vibe as another song.
- Small: the audio model is EfficientAT mn20_as (35 MB in 8-bit) and runs on a CPU; the search side is Qwen3-Embedding-0.6B plus a small trained layer (818 MB in 8-bit).
- Open: Apache-2.0, trained only on CC0, CC BY and public-domain music. Every training song is credited in
CREDITS.csv. - Languages: English, Spanish, German, French, Brazilian Portuguese and Japanese (translated training texts; scores per language are in
results/languages.md).
Try it: demo Β· benchmark: Divi19/vibebench
Use it
In AudioMuse-AI (self-hosted music server analysis): Plugins β add https://huggingface.co/Divi19/vibe-embed/resolve/main/plugin/manifest.json β install Vibe Search. The plugin indexes your library on your own machine; details in plugin/README.md.
In Python (onnxruntime β₯ 1.28 for the 8-bit search file; numpy; tokenizers):
from huggingface_hub import snapshot_download
import sys
d = snapshot_download("Divi19/vibe-embed")
sys.path.insert(0, d + "/plugin/plugins/VibeSearch")
import vibemodel # the plugin's loader: preprocessing plus self-checks against parity.json
m = vibemodel.VibeModel(d, "mn20_as", "int8")
m.load_audio(); m.load_text()
song_vec, n = m.embed_audio(wave_32k_mono) # float32 numpy array, 32 kHz mono
query_vecs = m.embed_texts(["rainy Sunday coffee shop"])
score = query_vecs @ song_vec # cosine similarity
The exact preprocessing is written in config.json: 32 kHz mono, 10-second windows (a last window of at least 2 s is kept), window vectors averaged and normalised; search texts get the instruction prompt in config.json, left padding, at most 256 tokens.
| file | what |
|---|---|
mn20_as/audio_encoder_int8.onnx, audio_encoder.onnx |
audio model, 8-bit and full precision (35 / 72 MB) |
text/text_encoder_int8.onnx, text_encoder.onnx (+ .data) |
search model, 8-bit and full precision (818 / 2396 MB) |
text/tokenizer/ |
tokenizer |
config.json, parity.json |
preprocessing, and reference outputs that loaders check against |
plugin/ |
the AudioMuse-AI plugin catalogue |
results/ |
full result tables |
Results
Measured by us, same code for every model. IncompeBench: 500 written searches over 1,574 permissively licensed songs, nDCG@10 Γ 100. Song Describer: one caption per recording, recall@10.
| model | size | licence | IncompeBench lenient | IncompeBench strict | Song Describer R@10 |
|---|---|---|---|---|---|
| vibe-embed v1.2, shipped 8-bit files (this repo) | 35 MB audio + 818 MB search | Apache-2.0 | 71.8 | 47.1 | |
| vibe-embed v1.2, full precision | 72 MB + 2396 MB | Apache-2.0 | 72 | 47.1 | |
| its teacher (LCO-Omni-7B + LoRA with e5-omni-7B, not released) | 2 Γ 7B | 79.2 | 58.8 | ||
| DCLAP (AudioMuse-AI's current model) | AGPL-3.0 | 56.2 | 46.5 | 30.8 | |
| LAION-CLAP music + speech | Apache-2.0 | 53.3 | 44.0 | 31.7 | |
| PE-AV large | Apache-2.0 | 55.9 | 47.8 | 48.6 | |
| PE-AV base | Apache-2.0 | 50.7 | 42.6 | 46.4 | |
| PE-AV small | Apache-2.0 | 52.8 | 44.6 | 47.8 | |
| LCO-Embedding-Omni-7B | Apache-2.0 | 76.9 | 68.2 | 41.9 | |
| e5-omni-7B | MIT | 74.4 | 65.3 | 53 |
Published on Mixedbread's IncompeBench page (tested March 2026; not re-measured by us):
| model | IncompeBench lenient | IncompeBench strict |
|---|---|---|
| Mixedbread (Wholembed V3, API only) | 77.3 | 65.1 |
| Gemini Embedding 2 (API only) | 72.5 | 64 |
| CLaMP 3 | 70.4 | 57.9 |
| ColQwen-Omni | 65.4 | 51.4 |
| TTMR++ | 64.5 | 50.2 |
| CLAP | 59.4 | 46 |
On IncompeBench (lenient) the shipped 8-bit files score 71.8: above CLaMP 3, ColQwen-Omni, TTMR++, CLAP; below Mixedbread (Wholembed V3, API only), Gemini Embedding 2 (API only) as published. The test half of IncompeBench, which was never used to choose between versions, gives 72.8 with the shipped 8-bit files.
VibeBench
VibeBench ranks 1,831 held-out CC songs for 500 real music requests from CPCD (Google Research, CC BY 4.0). Every system's top 10 is graded by an LLM (Phi-4) against a written description of each song. Main board: grades rescaled with cut-points fitted on IncompeBench's human-validated grades.
Rescaled grades (prompt-1 cut-points)
| system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 |
|---|---|---|---|---|
| teacher, its own 7B search side | 0.8031 | 0.7922 | 0.8382 | 1 |
| combined teacher, its own 7B search side | 0.7939 | 0.7866 | 0.8176 | 1 |
| combined teacher, the v1.2 search side | 0.7644 | 0.7537 | 0.7991 | 1 |
| teacher, the student's search side (Qwen3) | 0.7536 | 0.7397 | 0.7983 | 1 |
| LCO-Omni-7B as is | 0.7494 | 0.7415 | 0.775 | 1 |
| v1.2 student + v1.2 search side (ships) | 0.7329 | 0.7191 | 0.7775 | 1 |
| e5-omni-7B as is | 0.7309 | 0.7214 | 0.7615 | 1 |
| v1 student + v1.1 search side | 0.718 | 0.708 | 0.7507 | 1 |
| v1 student + v1 search side (07) | 0.6861 | 0.6769 | 0.7159 | 1 |
| DCLAP (AudioMuse-AI) | 0.6371 | 0.6312 | 0.6561 | 1 |
| LAION CLAP music+speech | 0.574 | 0.5683 | 0.5923 | 1 |
| BM25 over the song cards | 0.5307 | 0.5245 | 0.5508 | 1 |
As answered
| system | nDCG@10 all | nDCG@10 test | nDCG@10 dev | judged@10 |
|---|---|---|---|---|
| teacher, its own 7B search side | 0.8283 | 0.8238 | 0.8427 | 1 |
| combined teacher, its own 7B search side | 0.8132 | 0.8115 | 0.8186 | 1 |
| combined teacher, the v1.2 search side | 0.7847 | 0.782 | 0.7936 | 1 |
| teacher, the student's search side (Qwen3) | 0.7773 | 0.771 | 0.7977 | 1 |
| LCO-Omni-7B as is | 0.7635 | 0.7631 | 0.7649 | 1 |
| v1.2 student + v1.2 search side (ships) | 0.7509 | 0.7449 | 0.7705 | 1 |
| v1 student + v1.1 search side | 0.744 | 0.7391 | 0.76 | 1 |
| e5-omni-7B as is | 0.7421 | 0.7383 | 0.7545 | 1 |
| v1 student + v1 search side (07) | 0.7039 | 0.6994 | 0.7184 | 1 |
| DCLAP (AudioMuse-AI) | 0.6513 | 0.6472 | 0.6646 | 1 |
| LAION CLAP music+speech | 0.5857 | 0.5881 | 0.5779 | 1 |
| BM25 over the song cards | 0.5496 | 0.5486 | 0.5529 | 1 |
| v1.2 student + v1.2 search side minus | rescaled, all [95% CI] | rescaled, test [95% CI] | as answered, all [95% CI] |
|---|---|---|---|
| DCLAP (AudioMuse-AI) | +0.096 [+0.078, +0.115] | +0.088 [+0.067, +0.109] | +0.100 [+0.083, +0.117] |
| LAION CLAP music+speech | +0.159 [+0.139, +0.179] | +0.151 [+0.128, +0.175] | +0.165 [+0.147, +0.185] |
| v1 student + v1.1 search side | +0.015 [+0.004, +0.026] | +0.011 [-0.002, +0.025] | +0.007 [-0.002, +0.016] |
| v1 student + v1 search side (07) | +0.047 [+0.033, +0.060] | +0.042 [+0.028, +0.057] | +0.047 [+0.036, +0.059] |
| LCO-Omni-7B as is | -0.017 [-0.031, -0.002] | -0.022 [-0.040, -0.005] | -0.013 [-0.026, +0.001] |
| combined teacher, its own 7B search side | -0.061 [-0.074, -0.048] | -0.068 [-0.084, -0.052] | -0.062 [-0.074, -0.051] |
Speed
One 180-second song indexed on a Colab CPU (8 CPUs), decoding included, median of 3 runs. DCLAP is run exactly as AudioMuse-AI runs it.
| threads | ours int8 (s/song) | ours fp32 (s/song) | DCLAP (s/song) | DCLAP / ours int8 |
|---|---|---|---|---|
| 4 | 0.902 | 0.903 | 4.243 | 4.7 |
| 1 | 1.897 | 1.909 | 2.677 | 1.41 |
How it was made
- Captions. An audio-language model wrote listener-style descriptions of CC0 / CC BY Jamendo and FMA songs.
- Teacher. LCO-Embedding-Omni-7B, fine-tuned with LoRA on those pairs, combined with e5-omni-7B (both built on Qwen2.5-Omni-7B, Apache-2.0) and reduced to 512 numbers. The teacher is not released.
- Student. EfficientAT mn20_as (MIT; initialised from its authors' AudioSet weights) learns to reproduce the teacher's song vectors on 35,661 CC songs. The search side, Qwen3-Embedding-0.6B (Apache-2.0) with a trained layer, learns to reproduce the teacher's text vectors.
- Selection. Every choice between versions was made on the dev halves of IncompeBench and Song Describer; the test halves were only reported.
Never used for training: IncompeBench, Song Describer, MTG-Jamendo tags, or outputs of closed models.
Limits
- Trained and tested on royalty-free music. Mainstream commercial music was not part of training or testing, so how well it handles it is unmeasured.
- VibeBench is graded by an LLM. Its agreement with human-validated grades is ΞΊ β 0.47, below the usual 0.6 bar, so treat VibeBench as a secondary signal.
- Speed was measured on a cloud CPU, not on a Raspberry Pi.
- On onnxruntime older than 1.28 (for example AudioMuse-AI's no-AVX2 image), the 8-bit search file does not load; the plugin then uses the full-precision file, which is larger and needs more RAM. Results are the same.
Licence and credits
Model files: Apache-2.0 (LICENSE). The AudioMuse-AI plugin in plugin/ is AGPL-3.0, like AudioMuse-AI itself. Training music: CC0, CC BY and public-domain songs from Jamendo and the Free Music Archive; every song with its artist, licence and link is in CREDITS.csv (CC BY: 35,661).
- Downloads last month
- 27