Pixel-Linguist/Pixel-Linguist-II-Midtrain-Only

PIXEL LINGUIST II — Mid-Training Only

A vision-only encoder that reads, retrieves and compresses language directly in pixel space. Text is rendered to RGB images and encoded by the same tower that encodes natural images, so no text tokenizer is involved at inference time.

This is the "mid-training only" ablation from the paper — trained directly from the Qwen2.5-VL vision tower on the Stage 2 recipe (26M curated semantic text pairs + 26M LAION image-text pairs), skipping the Stage 1 multilingual Wikipedia pretraining. It is the checkpoint behind the paper's Table 3 and Table 4 headline rows.

Related releases:

Model	Curriculum
Pixel-Linguist-II-Pretrain	Stage 1 only
Pixel-Linguist-II-Midtrain	Stage 1 + Stage 2
this model	Stage 2 only

From the EMNLP paper On the Design Fundamentals of Pixel Text Representation Learning.

Model details

Architecture	Native-resolution ViT (NaViT-style), Qwen2_5_VisionTransformerPretrainedModel
Initialised from	Qwen2.5-VL vision tower
Parameters	676.6M
Embedding dim	3584
Precision	bfloat16
Input	Arbitrary resolution / aspect ratio, no lossy resizing
Curriculum	Stage 2 only (no Stage 1 pretraining)

Adjacent visual tokens are compressed by a 2x2 pooling layer; embeddings are mean-pooled over the (H/2)x(W/2) merged patches per image and L2-normalised.

Results

Spearman correlation for Visual STS, nDCG@5 for ViDoRe (6-subset average, the paper's convention). Qwen2.5-ViT is the untrained backbone.

Benchmark	Qwen2.5-ViT	Stage 1	Stage 2 only (this model)	Stage 1+2
Visual STS (English)	46.55	73.25	74.72	76.03
Visual STS (cross-lingual)	33.98	48.04	53.83	57.16
Visual STS (multilingual)	46.30	64.13	61.79	65.27
ViDoRe (6 subsets)	2.14	21.45	48.70	46.13

This checkpoint is the paper's strongest single model on ViDoRe and already sets state of the art on English Visual STS. Adding Stage 1 pretraining on top (pixel-linguist-2-midtrain) trades a little document retrieval for markedly better cross-lingual and multilingual alignment (+3.3 cross, +3.5 multi) — the curriculum finding of the paper's RQ4.

Usage

import json, torch, torch.nn.functional as F
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoImageProcessor
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VisionTransformerPretrainedModel
from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig

path = snapshot_download("Pixel-Linguist/Pixel-Linguist-II-Midtrain-Only")

config = Qwen2_5_VLVisionConfig(**json.load(open(f"{path}/config.json")))
model = Qwen2_5_VisionTransformerPretrainedModel(config)
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model = model.to("cuda", torch.bfloat16).eval()
processor = AutoImageProcessor.from_pretrained(path)


@torch.no_grad()
def encode(images):
    """Embed a list of PIL images (rendered text or natural photos)."""
    inputs = processor(images=images, return_tensors="pt")
    pixel_values = inputs["pixel_values"].to("cuda", torch.bfloat16)
    grid_thw = inputs["image_grid_thw"].to("cuda")

    hidden = model(hidden_states=pixel_values, grid_thw=grid_thw)

    sizes = ((grid_thw[:, 1] // 2) * (grid_thw[:, 2] // 2)).long()
    if sizes.sum() != hidden.shape[0]:
        sizes[-1] += hidden.shape[0] - sizes.sum()
    index = torch.repeat_interleave(torch.arange(len(grid_thw), device="cuda"), sizes)

    pooled = torch.zeros((len(grid_thw), hidden.shape[-1]), dtype=hidden.dtype, device="cuda")
    pooled.index_add_(0, index, hidden)
    pooled = pooled / sizes.unsqueeze(1).to(hidden.dtype).clamp(min=1)
    return F.normalize(pooled, dim=-1)

To embed a string, render it to an image first — that is the whole point of the model. Any renderer works; the paper samples from 393 fonts and 5,000+ DTD textured backgrounds during training so the encoder is robust to layout:

from PIL import Image, ImageDraw

def render(text, size=(448, 64)):
    img = Image.new("RGB", size, "white")
    ImageDraw.Draw(img).text((5, 25), text, fill="black")
    return img

emb = encode([render("a dog runs in the park"),
              render("a puppy is running outside"),
              render("quantum field theory")])
emb @ emb.T

Reproducibility notes. transformers>=4.57 loads the image processor as Qwen2VLImageProcessorFast by default and warns that this "may produce slightly different outputs"; pass use_fast=False to AutoImageProcessor.from_pretrained if you need the slow processor. Scores are also sensitive to how text is rendered — the canvas size and font size act as spatial proxies for this model, so the same string rendered at a different width will give a different embedding.

Training

Text corpus	26M curated multilingual semantic text pairs
Image corpus	26M natural image-text pairs from LAION-2B
Init from	Qwen2.5-VL vision tower (no Stage 1)
Objective	Symmetric in-batch InfoNCE, embeddings all-gathered across ranks, logit_scale = 1/0.03
Epochs	2
LR / batch	5e-5 / 1024
Infra	64 GPUs, bf16, DeepSpeed ZeRO-2, gradient checkpointing

Text is rendered on the fly with randomised fonts, sizes, backgrounds, brightness and blur, so the model never sees the same visual instantiation of a string twice. Natural image-text pairs act as a required regulariser — dropping them collapses document retrieval even at full data scale.

Citation

@inproceedings{yuan2026pixel,
  title     = {On the Design Fundamentals of Pixel Text Representation Learning},
  author    = {Yuan, Chaohao and Yuan, Ruifeng and Huang, Zhuoxu and Rong, Yu and
               Cheng, Hong and Chan, Hou Pong and Xiao, Chenghao},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}