ColUltraVBERT

ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from UltraVBERT. It encodes every query token and every page token as a 768-d vector and scores a page against a query with MaxSim. The first 128, 256, 320 or 512 dimensions of each vector form a smaller index. Try it in the ColUltraVBERT Space.

Models

Model	Role	Status
UltraVBERT	Base encoder aligned to page images	Available
BiUltraVBERT	Single-vector retriever for visual documents	Cooking
ColUltraVBERT	Late-interaction retriever for visual documents	Available · Demo

Data

Dataset	Content
VDR Train 500k	499,704 query-page pairs; queries in English (73%), Spanish, German, French, Italian and Portuguese

Usage

Requires sentence-transformers>=6.1 and transformers>=5.20.

from PIL import Image
from sentence_transformers import MultiVectorEncoder

model = MultiVectorEncoder("athrael-soju/ColUltraVBERT", trust_remote_code=True)

pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]

page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)

# One row per query, one column per page.
scores = model.similarity(query_embeddings, page_embeddings)

For a smaller index, keep the first 128, 256, 320 or 512 dimensions of every vector and L2-normalize them again; the tables below score each size.

Model

A learned linear layer maps each 768-d token state from UltraVBERT to a 768-d retrieval vector, trained with Matryoshka prefixes so the first 128, 256, 320 and 512 dimensions also work on their own. Each token vector is L2-normalized and padding tokens are dropped. The image processor resizes a page to a 3,072 px longest edge and splits it into up to 27 tiles of 512 px plus a downscaled copy of the whole page, so a page yields at most 1,792 image-token vectors.

Training

Four fine-tunes of UltraVBERT on VDR Train 500k, differing in hard negatives, distillation and seed, are merged layer by layer into the released weights. Each fine-tune trains LoRA adapters (rank 32) on the text encoder and the output projection for one epoch at batch 256; the vision encoder and connector stay frozen and shared.

ViDoRe

Scores from MTEB 2.26.3, in bf16 with SDPA attention, at 768 dimensions. A task with several language splits reports their mean. ColModernVBERT's scores are its published MTEB results.

ViDoRe V1 (nDCG@5)	ColModernVBERT	ColUltraVBERT
ArxivQA	73.2	80.3
DocVQA	55.6	57.7
InfoVQA	86.4	89.5
TabFQuAD	74.8	92.5
TAT-DQA	73.1	75.1
ShiftProject	60.4	81.3
SyntheticDocQA AI	97.9	99.6
SyntheticDocQA Energy	93.3	95.3
SyntheticDocQA Government Reports	92.3	95.1
SyntheticDocQA Healthcare Industry	95.3	97.9
Mean	80.2	86.4

ViDoRe V2 (nDCG@5)	ColModernVBERT	ColUltraVBERT
ESG Reports	40.1	49.7
Economics Reports	29.2	51.5
BioMedical Lectures	31.1	54.3
ESG Reports HL	52.2	63.6
Mean	38.2	54.8

ViDoRe V3 (nDCG@10)	ColModernVBERT	ColUltraVBERT
Computer Science	37.6	66.5
Energy	32.5	51.9
Finance EN	28.8	53.4
Finance FR	19.8	33.3
HR	19.8	51.2
Industrial	15.7	37.7
Pharmaceuticals	32.2	57.6
Physics	22.6	37.9
Mean	26.1	48.7

Smaller indexes

Means at each vector size and page budget over the same ViDoRe tasks, scored on cached page features, which agree with MTEB within 0.0035 nDCG per task. The 2,048 px, 16-tile budget sets size.longest_edge to 2048 and max_image_tiles to 16 in processor_config.json and caps a page at 1,088 vectors.

Vector size	Pages	V1	V2	V3
768	3,072 px, 27 tiles	86.4	54.8	48.7
512	3,072 px, 27 tiles	86.3	54.3	48.4
320	3,072 px, 27 tiles	86.0	54.2	48.2
256	3,072 px, 27 tiles	86.0	54.3	48.0
128	3,072 px, 27 tiles	85.7	54.3	46.8
768	2,048 px, 16 tiles	86.3	53.3	47.6
512	2,048 px, 16 tiles	86.3	52.9	47.5
320	2,048 px, 16 tiles	86.4	52.4	47.2
256	2,048 px, 16 tiles	86.4	52.1	46.9
128	2,048 px, 16 tiles	85.6	52.6	45.9