OpenSearch-AI

/

Ops-Colqwen3-4B

like 11

Follow

OpenSearch-AI 40

Hello!

I recently released the new MultiVectorEncoder class in Sentence Transformers v6.0, and I think this model would be a great fit for it. You can read more about the release here: https://huggingface.co/blog/multi-vector-encoder. I would also love to feature this model in the accompanying documentation.

Heads up, this PR was AI-generated and human-reviewed. Here's a summary of the changes as reported by my agent:

Pull Request overview

• Integrate OpenSearch-AI/Ops-Colqwen3-4B with Sentence Transformers as a multi-vector (ColBERT-style late interaction) retriever via MultiVectorEncoder.
• Fix the custom processor and modeling code for transformers 5.x compatibility.
• Add a Sentence Transformers usage section to the model card, plus the sentence-transformers and multi-vector tags.

Details

The integration keeps the repository's own custom code in charge: Sentence Transformers loads the model with the same trust_remote_code=True it already requires and drives the repository's own embedding forward directly, so the exact embeddings the modeling code produces (already projected and normalized) are used as is. The pipeline is just two modules, the transformer plus a MultiVectorMask that drops padding tokens. A named chat template in additional_chat_templates/sentence_transformers.jinja reproduces the query and image prompt formats from the processor, so no manual prefixing is needed. The weights are untouched.

Two of the changes double as bug fixes for plain transformers users on 5.x. First, the processor gains a standard __call__ (previously only process_queries and process_images existed, so AutoProcessor instances could not be called directly): it routes text to the query format and images to the document format, and dedupes padding kwargs that 5.x delivers both flat and nested. Second, the model forward now passes mm_token_type_ids through to the backbone. Qwen3-VL's M-RoPE requires it on transformers 5.x, and without it image inputs fail on current versions.

Since config.json declares float32, a plain load runs in fp32. The snippet below passes model_kwargs={"dtype": "bfloat16"} to match the model card's own usage. I did not change the config default, that call is yours to make.

Verified against the repository's own pipeline on text queries and image documents: the per-token embeddings match with cosine similarity 1.0000 and the MaxSim scores differ by at most 0.0025. For reference, on a 400-query NanoViDoRe v3 subset this integration scores 0.6150 nDCG@10, consistent with the model's 0.6095 ViDoRe v3 leaderboard entry.

pip install "sentence-transformers[image]>=6.0.0"

from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder(
    "OpenSearch-AI/Ops-Colqwen3-4B",
    trust_remote_code=True,
    model_kwargs={"dtype": "bfloat16"},
)

queries = [
    "What is the variable represented on the y-axis of the graph?",
    "Total outlay is maximum in which year?",
]
images = [
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
    "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
]

query_embeddings = model.encode_query(queries)
image_embeddings = model.encode_document(images)
print(query_embeddings[0].shape, image_embeddings[0].shape)
# torch.Size([25, 2560]) torch.Size([1254, 2560])

# Diagonal should have higher scores
scores = model.similarity(query_embeddings, image_embeddings)
print(scores)
# tensor([[17.4668, 12.9785],
#         [ 7.0088, 15.9492]], device='cuda:0')

To try this before merging, pass revision="refs/pr/5" to MultiVectorEncoder.

Happy to tweak anything you'd like changed. Please let me know if you have any questions or feedback!

• Tom Aarsen