vibe-embed v1.2

Search music by how it sounds and feels: "rainy Sunday coffee shop", "upbeat but melancholic, no vocals", "something for a long night drive". Songs and descriptions share one 512-number vector space, so you can search a library by text or find songs with the same vibe as another song.

  • Small: the audio model is EfficientAT mn20_as (35 MB in 8-bit) and runs on a CPU; the search side is Qwen3-Embedding-0.6B plus a small trained layer (818 MB in 8-bit).
  • Open: Apache-2.0, trained only on CC0, CC BY and public-domain music. Every training song is credited in CREDITS.csv.
  • Languages: English, Spanish, German, French, Brazilian Portuguese and Japanese (translated training texts; scores per language are in results/languages.md).

Try it: demo Β· benchmark: Divi19/vibebench

Use it

In AudioMuse-AI (self-hosted music server analysis): Plugins β†’ add https://huggingface.co/Divi19/vibe-embed/resolve/main/plugin/manifest.json β†’ install Vibe Search. The plugin indexes your library on your own machine; details in plugin/README.md.

In Python (onnxruntime β‰₯ 1.28 for the 8-bit search file; numpy; tokenizers):

from huggingface_hub import snapshot_download
import sys
d = snapshot_download("Divi19/vibe-embed")
sys.path.insert(0, d + "/plugin/plugins/VibeSearch")
import vibemodel                                   # the plugin's loader: preprocessing plus self-checks against parity.json
m = vibemodel.VibeModel(d, "mn20_as", "int8")
m.load_audio(); m.load_text()
song_vec, n = m.embed_audio(wave_32k_mono)          # float32 numpy array, 32 kHz mono
query_vecs = m.embed_texts(["rainy Sunday coffee shop"])
score = query_vecs @ song_vec                       # cosine similarity

The exact preprocessing is written in config.json: 32 kHz mono, 10-second windows (a last window of at least 2 s is kept), window vectors averaged and normalised; search texts get the instruction prompt in config.json, left padding, at most 256 tokens.

file what
mn20_as/audio_encoder_int8.onnx, audio_encoder.onnx audio model, 8-bit and full precision (35 / 72 MB)
text/text_encoder_int8.onnx, text_encoder.onnx (+ .data) search model, 8-bit and full precision (818 / 2396 MB)
text/tokenizer/ tokenizer
config.json, parity.json preprocessing, and reference outputs that loaders check against
plugin/ the AudioMuse-AI plugin catalogue
results/ full result tables

Results

Measured by us, same code for every model. IncompeBench: 500 written searches over 1,574 permissively licensed songs, nDCG@10 Γ— 100. Song Describer: one caption per recording, recall@10.

model size licence IncompeBench lenient IncompeBench strict Song Describer R@10
vibe-embed v1.2, shipped 8-bit files (this repo) 35 MB audio + 818 MB search Apache-2.0 71.8 47.1
vibe-embed v1.2, full precision 72 MB + 2396 MB Apache-2.0 72 47.1
its teacher (LCO-Omni-7B + LoRA with e5-omni-7B, not released) 2 Γ— 7B 79.2 58.8
DCLAP (AudioMuse-AI's current model) AGPL-3.0 56.2 46.5 30.8
LAION-CLAP music + speech Apache-2.0 53.3 44.0 31.7
PE-AV large Apache-2.0 55.9 47.8 48.6
PE-AV base Apache-2.0 50.7 42.6 46.4
PE-AV small Apache-2.0 52.8 44.6 47.8
LCO-Embedding-Omni-7B Apache-2.0 76.9 68.2 41.9
e5-omni-7B MIT 74.4 65.3 53

Published on Mixedbread's IncompeBench page (tested March 2026; not re-measured by us):

model IncompeBench lenient IncompeBench strict
Mixedbread (Wholembed V3, API only) 77.3 65.1
Gemini Embedding 2 (API only) 72.5 64
CLaMP 3 70.4 57.9
ColQwen-Omni 65.4 51.4
TTMR++ 64.5 50.2
CLAP 59.4 46

On IncompeBench (lenient) the shipped 8-bit files score 71.8: above CLaMP 3, ColQwen-Omni, TTMR++, CLAP; below Mixedbread (Wholembed V3, API only), Gemini Embedding 2 (API only) as published. The test half of IncompeBench, which was never used to choose between versions, gives 72.8 with the shipped 8-bit files.

VibeBench

VibeBench ranks 1,831 held-out CC songs for 500 real music requests from CPCD (Google Research, CC BY 4.0). Every system's top 10 is graded by an LLM (Phi-4) against a written description of each song. Main board: grades rescaled with cut-points fitted on IncompeBench's human-validated grades.

Rescaled grades (prompt-1 cut-points)

system nDCG@10 all nDCG@10 test nDCG@10 dev judged@10
teacher, its own 7B search side 0.8031 0.7922 0.8382 1
combined teacher, its own 7B search side 0.7939 0.7866 0.8176 1
combined teacher, the v1.2 search side 0.7644 0.7537 0.7991 1
teacher, the student's search side (Qwen3) 0.7536 0.7397 0.7983 1
LCO-Omni-7B as is 0.7494 0.7415 0.775 1
v1.2 student + v1.2 search side (ships) 0.7329 0.7191 0.7775 1
e5-omni-7B as is 0.7309 0.7214 0.7615 1
v1 student + v1.1 search side 0.718 0.708 0.7507 1
v1 student + v1 search side (07) 0.6861 0.6769 0.7159 1
DCLAP (AudioMuse-AI) 0.6371 0.6312 0.6561 1
LAION CLAP music+speech 0.574 0.5683 0.5923 1
BM25 over the song cards 0.5307 0.5245 0.5508 1

As answered

system nDCG@10 all nDCG@10 test nDCG@10 dev judged@10
teacher, its own 7B search side 0.8283 0.8238 0.8427 1
combined teacher, its own 7B search side 0.8132 0.8115 0.8186 1
combined teacher, the v1.2 search side 0.7847 0.782 0.7936 1
teacher, the student's search side (Qwen3) 0.7773 0.771 0.7977 1
LCO-Omni-7B as is 0.7635 0.7631 0.7649 1
v1.2 student + v1.2 search side (ships) 0.7509 0.7449 0.7705 1
v1 student + v1.1 search side 0.744 0.7391 0.76 1
e5-omni-7B as is 0.7421 0.7383 0.7545 1
v1 student + v1 search side (07) 0.7039 0.6994 0.7184 1
DCLAP (AudioMuse-AI) 0.6513 0.6472 0.6646 1
LAION CLAP music+speech 0.5857 0.5881 0.5779 1
BM25 over the song cards 0.5496 0.5486 0.5529 1
v1.2 student + v1.2 search side minus rescaled, all [95% CI] rescaled, test [95% CI] as answered, all [95% CI]
DCLAP (AudioMuse-AI) +0.096 [+0.078, +0.115] +0.088 [+0.067, +0.109] +0.100 [+0.083, +0.117]
LAION CLAP music+speech +0.159 [+0.139, +0.179] +0.151 [+0.128, +0.175] +0.165 [+0.147, +0.185]
v1 student + v1.1 search side +0.015 [+0.004, +0.026] +0.011 [-0.002, +0.025] +0.007 [-0.002, +0.016]
v1 student + v1 search side (07) +0.047 [+0.033, +0.060] +0.042 [+0.028, +0.057] +0.047 [+0.036, +0.059]
LCO-Omni-7B as is -0.017 [-0.031, -0.002] -0.022 [-0.040, -0.005] -0.013 [-0.026, +0.001]
combined teacher, its own 7B search side -0.061 [-0.074, -0.048] -0.068 [-0.084, -0.052] -0.062 [-0.074, -0.051]

Speed

One 180-second song indexed on a Colab CPU (8 CPUs), decoding included, median of 3 runs. DCLAP is run exactly as AudioMuse-AI runs it.

threads ours int8 (s/song) ours fp32 (s/song) DCLAP (s/song) DCLAP / ours int8
4 0.902 0.903 4.243 4.7
1 1.897 1.909 2.677 1.41

How it was made

  1. Captions. An audio-language model wrote listener-style descriptions of CC0 / CC BY Jamendo and FMA songs.
  2. Teacher. LCO-Embedding-Omni-7B, fine-tuned with LoRA on those pairs, combined with e5-omni-7B (both built on Qwen2.5-Omni-7B, Apache-2.0) and reduced to 512 numbers. The teacher is not released.
  3. Student. EfficientAT mn20_as (MIT; initialised from its authors' AudioSet weights) learns to reproduce the teacher's song vectors on 35,661 CC songs. The search side, Qwen3-Embedding-0.6B (Apache-2.0) with a trained layer, learns to reproduce the teacher's text vectors.
  4. Selection. Every choice between versions was made on the dev halves of IncompeBench and Song Describer; the test halves were only reported.

Never used for training: IncompeBench, Song Describer, MTG-Jamendo tags, or outputs of closed models.

Limits

  • Trained and tested on royalty-free music. Mainstream commercial music was not part of training or testing, so how well it handles it is unmeasured.
  • VibeBench is graded by an LLM. Its agreement with human-validated grades is ΞΊ β‰ˆ 0.47, below the usual 0.6 bar, so treat VibeBench as a secondary signal.
  • Speed was measured on a cloud CPU, not on a Raspberry Pi.
  • On onnxruntime older than 1.28 (for example AudioMuse-AI's no-AVX2 image), the 8-bit search file does not load; the plugin then uses the full-precision file, which is larger and needs more RAM. Results are the same.

Licence and credits

Model files: Apache-2.0 (LICENSE). The AudioMuse-AI plugin in plugin/ is AGPL-3.0, like AudioMuse-AI itself. Training music: CC0, CC BY and public-domain songs from Jamendo and the Free Music Archive; every song with its artist, licence and link is in CREDITS.csv (CC BY: 35,661).

Downloads last month
27
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Divi19/vibe-embed

Quantized
(285)
this model

Dataset used to train Divi19/vibe-embed

Space using Divi19/vibe-embed 1