Sentio

Sentio is WeMM-Embedding-9B with ears, and with sharper eyes for video moments.

WeMM-Embedding-9B is the strongest open embedding model for text, images, video and visual documents, but it cannot hear. Wianmu-5B gave the 4B member of the family an audio tower by freezing both the embedding model and a pretrained audio encoder and training only a small connector between them, the recipe of Tianmu-Emb-Uni. Sentio does the same to the 9B, with two additions: before the audio stage, the 9B tower gets a small LoRA fine-tune for video moment retrieval, so the vision side also moves; and during the audio stage the audio encoder gets a LoRA of its own, trained together with the connector. The name is Latin for "I perceive": one model that hears, sees and reads into one space. Total 10.20B parameters, of which 152M (1.5%) were trained.

How it works

Two experts, one that understands audio and one that understands everything else, and a trained piece in the middle that moves audio into the other model's embedding space.

The audio side is the audio tower of Qwen2.5-Omni-3B (the encoder only, 638M parameters) with a LoRA on its attention and MLP projections. Everything else goes through WeMM-Embedding-9B. Audio lands in WeMM's 4096-d space, so an audio clip can be compared against a caption, a label name, an image or a video directly, and nothing that was indexed with the tower has to be re-indexed.

waveform
  -> Whisper-style mel features (128 bins, up to 30 s)
  -> Qwen2.5-Omni audio tower + LoRA (r=16), masked-mean pooled over frames  -> 2048-d
  -> connector: 3-layer MLP, 2048 -> 4096 -> 8192 -> 4096, LayerNorm + GELU
  -> modality adapter fusion
  -> RMSNorm projection
  -> L2 normalize  -> 4096-d embedding

Two stages, three trained pieces.

1. Stage 1, the tower. WeMM-Embedding-9B gets a LoRA (r=16, α=32, 43.3M parameters) trained with InfoNCE on video moment-retrieval pairs mixed with a replay of visual-document, video and image pairs so that nothing already strong is lost. This is the only change to the non-audio path: a text, image, video or document embedding from Sentio is the embedding of WeMM-9B with this LoRA applied. It lifts MMEB-V2 from 80.58 to 80.68 and V-MRET from 58.46 to 61.53.
2. Stage 2, the ears. With the tower frozen (LoRA included), the audio path is trained: a LoRA on the audio tower (r=16, α=32, dropout 0.05, on q/k/v/out projections and both MLP layers of every encoder block, 11.8M parameters) together with the audio head — connector, adapter fusion, projection norm and prototypes, 96.8M parameters, almost all of it the connector (75.5M).

Adapter fusion. One audio clip can be matched against very different things. For a caption you care about meaning, for another audio clip about acoustic detail, for a video about timing, and for a label about the category. So there are four small bottleneck adapters, one each for text, audio, video and label. A gate looks at the audio vector and mixes the four outputs with soft weights, with a small entropy bonus during training so the gate doesn't collapse onto one adapter.

Adapter	Input	Down projection	Activation	Bottleneck	Up projection	Output	Residual	Parameters
text	4096	Linear 4096 → 512, no bias	GELU	512	Linear 512 → 4096, no bias	4096	yes	4,194,304
audio	4096	Linear 4096 → 512, no bias	GELU	512	Linear 512 → 4096, no bias	4096	yes	4,194,304
video	4096	Linear 4096 → 512, no bias	GELU	512	Linear 512 → 4096, no bias	4096	yes	4,194,304
label	4096	Linear 4096 → 512, no bias	GELU	512	Linear 512 → 4096, no bias	4096	yes	4,194,304
gate	4096	Linear 4096 → 1024, no bias + LayerNorm(1024)	GELU	1024	Linear 1024 → 4, with bias	4 (softmax weights)	no	4,200,452

Shared prototypes. 64 learnable vectors in the shared space. Each audio embedding and each target embedding gets a soft assignment over them, and training pushes the audio and its matching target to pick the same prototypes. They are only used for the loss; at inference you just use the embedding.

Parameter count (measured from the weight files):

Component	Parameters	Trained
WeMM-Embedding-9B tower	9,408M	frozen
vdr LoRA on the tower (stage 1)	43.3M	yes
Qwen2.5-Omni-3B audio tower	638M	frozen
LoRA on the audio tower (stage 2)	11.8M	yes
Connector (3-layer MLP)	75.5M	yes (stage 2)
Adapter fusion (4 adapters + gate)	21.0M	yes (stage 2)
RMSNorm projection + 64 prototypes	0.27M	yes (stage 2)
Total	10.20B	152M (1.5%)

Training

Stage 2 (audio). The loss is symmetric in-batch InfoNCE between audio and target embeddings (temperature 0.07), which does most of the work. On top of it come the Tianmu prototype loss — a contrastive term over prototype assignments (0.03), a symmetric KL between the audio and target assignments (0.01) and a balance term so all prototypes get used (0.005) — and a gate-entropy bonus (0.01). For label datasets, batches are built so that each label appears at most once; otherwise two clips of "dog bark" in one batch count as negatives of each other.

The data is 449k audio–target pairs over 37k distinct targets from seven public datasets, sampled in natural proportion: NSynth (instrument labels, the bulk of every epoch), SpeechCommands (spoken-word labels), FSD50K and RAVDESS (sound-event and emotion labels), Clotho (captions), Flickr8k Audio (spoken captions paired with the image they describe) and AVE (clips paired with their source video). Batch size 16 (smaller where a dataset has fewer distinct labels); learning rate 1e-5 for the head and 1e-4 for the audio-tower LoRA, constant after 500 warm-up steps; 498,000 steps on one H200. Validation is the same in-batch InfoNCE on a held-out split every 2,500 steps; the released checkpoint is the one with the lowest validation loss (step ~408,000).

Stage 1 (tower). LoRA r=16, α=32 on the WeMM-9B tower, 600 steps of InfoNCE at temperature 0.045; 8/11 of each batch are DiDeMo moment–query pairs and 1/11 each MSR-VTT video–text, MSCOCO text-to-image and ColPali page–query pairs, replayed so that video retrieval, image retrieval and visual documents keep their scores while V-MRET moves.

Evaluation

All 190 MMEB-V3 tasks with the Tencent WeMM evaluation harness, with the Wianmu audio modifications for the 11 audio tasks; scored with the leaderboard's own code. For the two TUTSound tasks, whose queries carry both an audio clip and text, the model has no joint encoder, so the two unit embeddings are summed with equal weight and re-normalised.

MMEB-V3, leaderboard snapshot of 16 Oct 2026:

Model	Overall	Overall-V3 (new)	Text	Audio	Agent	Image	Video	Visdoc
Sentio	61.69	48.46	48.98	36.81	50.57	80.90	75.04	83.28
Wianmu-5B-v0	60.45	47.50	48.13	42.93	48.00	79.73	71.56	81.99
WeMM-Embedding-9B	59.33	44.54	48.76	—	50.10	80.99	74.31	83.34
Wianmu-3B-v0	58.16	44.37	45.30	41.16	44.11	78.67	70.79	80.80

MMEB-V2 (78 tasks) and the video block:

Model	Overall-V2	Image (V2)	Video	V-CLS	V-QA	V-RET	V-MRET	Visdoc
Sentio	80.68	81.77	75.04	87.54	77.89	67.80	61.53	83.28
WeMM-Embedding-9B	80.58	81.87	74.31	87.36	77.69	67.38	58.46	83.34

Audio tasks (hit@1):

Task	Sentio	Wianmu-3B-v0	Wianmu-5B-v0
NSynth	76.10	74.10	60.90
ESC-50	56.15	79.20	86.60
UrbanSound8K	55.92	64.05	66.63
SpeechCommands	89.50	89.80	89.90
CREMA-D	33.74	47.08	48.93
Clotho	8.48	13.61	16.56
SoundDescs	6.10	9.50	20.50
TUTSound	61.76	52.96	46.89
TUTSound (hard)	5.16	4.55	3.64
AVE	9.20	16.42	18.66
SpeechCOCO	2.80	1.50	13.00
A-CLS / A-RET / Audio-Overall	62.28 / 15.58 / 36.81	70.85 / 16.42 / 41.16	70.59 / 19.88 / 42.93

Sentio takes the top of the 190-task board on the strength of the 9B tower (text, agent, video and visual documents) and is the first of a series: the audio block is where it trails the smaller Wianmu models, and the next releases work on exactly that.

Known limitations

• Audio is pooled into a single vector, so there is no fine timing inside a clip. Grounding and moment-level tasks suffer from this the most.
• Clips longer than 30 seconds get cut.
• The audio stage saw seven datasets, dominated by NSynth instrument labels. Together with the adaptable encoder this makes the model strong on instruments and sound events (NSynth, TUTSound) and weaker than the smaller Wianmu models on environmental scenes (ESC-50, UrbanSound8K), emotion (CREMA-D) and audio captions (Clotho, SoundDescs).
• The tower is frozen during the audio stage, so whatever WeMM-9B can't tell apart in text, Sentio can't either. Some label names also collide across datasets ("happy" is a spoken word in SpeechCommands and an emotion in RAVDESS, and both share one target vector).
• Most of the training targets are English.

Usage

The repository holds the model card, config.json and modeling_sentio.py; the base checkpoints (WeMM-Embedding-9B, Qwen2.5-Omni-3B) are downloaded from their own repositories, and Sentio's trained parts (the vdr LoRA adapter, 43M, and trainable.pt with the audio head and the audio-tower LoRA, 109M) are not distributed here.

from modeling_sentio import Sentio

model = Sentio.from_pretrained(
    ".",
    wemm_path="path/to/WeMM-Embedding-9B",
    wemm_adapter_path="path/to/sentio/vdr_lora_adapter",
    qwen_omni_path="path/to/Qwen2.5-Omni-3B",
    trainable_path="path/to/sentio/trainable.pt",
)
audio = model.encode_audio(["dog.wav", "piano.wav"])          # (2, 4096)
text  = model.encode_text(["dog barking", "piano"])            # (2, 4096)
image = model.encode_image(["photo.jpg"])                      # (1, 4096)
scores = audio @ text.T

Preprocessing (128-bin mel features, 30-second head crop, mean pooling over valid frames) mirrors the MMEB-V3 evaluation that produced the scores above.

License

Sentio's own code, model card and trained parameters are released under Apache-2.0. The two base models keep their own terms: WeMM-Embedding-9B is licensed by Tencent under Apache-2.0; the audio encoder is the audio tower of Qwen2.5-Omni-3B, distributed by Alibaba Cloud under the Qwen Research License, whose terms apply to that component.

Credits

Credits to Wianmu-5B, Tianmu-Emb-Uni and the WeMM-Embedding family.