OmniUE-7B

OmniUE is an instruction-conditioned universal embedder that maps text, images, video, and audio into one 3,584-dimensional embedding space. It also supports visual interaction through bounding boxes or points, and audio interaction through time spans.

The checkpoint is a LoRA adapter on Qwen2.5-Omni-7B, with a SAM-Audio + SAM3 interaction module.

Resources

• Paper
• Webpage

Experimental Results

Best score in each column is highlighted. Rows tagged Ours are OmniUE.

MMEB-v2-video — text-interactive video

Gains of OmniUE-3B / 7B over the best baseline of the same size: overall +5.1 / +4.3.

MAEB — text-interactive audio

OmniUE-7B is best overall (53.4). Against LCO-Emb-7B it leads 7 of 8 meta-tasks. Clustering stays higher for Qwen2-Audio and the CLAP models, and cross-modal retrieval stays higher for LCO-Emb.

SCaR — visual-interactive text-image-to-text

Overall gain over the best 2B / 7B baseline: +20.7 / +27.4 points. The task is region-level caption retrieval from an image, a text instruction, and a box.

OmniCHOIR — omni-interactive audio retrieval

Recall@1. For baselines that cannot take a span, the pair is “span written as text / audio cropped to the span.” Masks and spans each beat text-only prompting, and using all three prompts is best (28.4).

Installation

Use a CUDA-enabled Python environment with Python 3.10 or newer:

git clone https://huggingface.co/Sony/OmniUE-7B
cd OmniUE-7B
pip install -e .

ffmpeg must be available on PATH to decode audio from video files.

The OmniUE adapter, SAM-Audio checkpoint, and SAM3 checkpoint are bundled in this repository. On first use, Hugging Face downloads the Qwen2.5-Omni-7B and T5 base-model dependencies automatically.

Quick start

from omniue import OmniUE

model = OmniUE.from_pretrained(".")

captions = [
    "A person plays a guitar on a stage.",
    "A dog runs across a field.",
    "A chef chops vegetables in a kitchen.",
]
caption_embeddings = model.encode_text(captions)

query_embedding = model.encode_video(
    "concert.mp4",
    instruction="Find the matching description.",
)
indices, scores = model.cosine_topk(query_embedding, caption_embeddings, k=3)

for index, score in zip(indices[0], scores[0]):
    print(f"{score:.4f}  {captions[index]}")

Run the same text/video retrieval example from the command line:

python examples/retrieve.py --video concert.mp4

API

Every encoder returns a float32 NumPy array with shape [batch_size, 3584]. The output is L2-normalized.

text = model.encode_text(["a dog", "a cat"], instruction="")
image = model.encode_image("image.jpg", instruction="Find the main subject.")
video = model.encode_video("video.mp4", instruction="Represent this video.")
audio = model.encode_audio("audio.wav", instruction="Represent this sound.")

Use a visual prompt to embed a selected object:

selected_object = model.encode_image(
    "image.jpg",
    instruction="Represent the selected object.",
    bbox=[0.1, 0.1, 0.9, 0.9],
)

Use a temporal audio prompt to embed a selected sound:

selected_sound = model.encode_audio(
    "audio.wav",
    instruction="Represent the selected sound.",
    span=(3.0, 8.0),
)

bbox and point coordinates are normalized to [0, 1]. Bounding boxes use [x1, y1, x2, y2]. Point prompts use points=[[x, y], ...] and point_labels=[1, 0, ...], where 1 denotes foreground and 0 background.

Run the lightweight package check:

python examples/smoke_test.py

Use --with-model to exercise text, image, audio, and video inference. Add --with-interactions to test visual and audio interaction paths.

Limitations and intended use

This release is intended for research on multimodal retrieval and interactive embedding. It has not been evaluated for high-stakes or safety-critical decisions. Performance can vary by domain, modality quality, language, and prompt phrasing. Users are responsible for respecting the licenses and acceptable-use terms of Qwen2.5-Omni, SAM-Audio, SAM3, and any input data.

Licenses

This repository contains components under different licenses:

• The OmniUE-specific checkpoint weights and original inference wrapper are released under CC BY-NC 4.0, as described in the root LICENSE.
• The bundled SAM-Audio code and checkpoint retain Meta's SAM License in sam-audio/LICENSE.
• The bundled SAM3 code and sam3.pt retain Meta's SAM License in SAM3_LICENSE.
• Qwen-derived source files retain their upstream notices and licenses. Qwen2.5-Omni itself is downloaded from its upstream repository and is not relicensed by this repository.

The CC BY-NC 4.0 license does not apply to the bundled third-party materials.

Citation

@inproceedings{wang2026omniue,
  title     = {Omni-Interactive Universal Embedder},
  author    = {Wang, Wei-Yao and Tateishi, Kazuya and Cui, Shuyang and
               Simon, Christian and Shibuya, Takashi and Takahashi, Shusuke and
               Mitsufuji, Yuki},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}