Pixel-Linguist/Pixel-Linguist-II-Pretrain

PIXEL LINGUIST II — Stage 1 (Foundational Pretraining)

A vision-only encoder that reads, retrieves and compresses language directly in pixel space. Text is rendered to RGB images and encoded by the same tower that encodes natural images, so no text tokenizer is involved at inference time.

This repository is the Stage 1 checkpoint — the end of foundational pretraining on 62M multilingual Wikipedia documents plus 26M LAION-2B image-text pairs.

Related releases:

Model	Curriculum
this model	Stage 1 only
Pixel-Linguist-II-Midtrain	Stage 1 + Stage 2
Pixel-Linguist-II-Midtrain-Only	Stage 2 only

From the EMNLP paper On the Design Fundamentals of Pixel Text Representation Learning.

Model details

Architecture	Native-resolution ViT (NaViT-style), Qwen2_5_VisionTransformerPretrainedModel
Initialised from	Qwen2.5-VL vision tower
Parameters	676.6M
Embedding dim	3584
Precision	bfloat16
Input	Arbitrary resolution / aspect ratio, no lossy resizing
Stage	1 of 2 (pretraining only)

Adjacent visual tokens are compressed by a 2x2 pooling layer; embeddings are mean-pooled over the (H/2)x(W/2) merged patches per image and L2-normalised.

Results

Spearman correlation for Visual STS, nDCG@5 for ViDoRe. Qwen2.5-ViT is the untrained backbone this model starts from.

Benchmark	Qwen2.5-ViT	Stage 1 (this model)	Stage 1+2
Visual STS (English)	46.55	73.25	76.03
Visual STS (cross-lingual)	33.98	48.04	57.16
Visual STS (multilingual)	46.30	64.13	65.27
ViDoRe (6 subsets)	2.14	21.45	46.13

Stage 1 already lifts English Visual STS by ~27 points over the backbone. Cross-lingual and document-retrieval ability, however, is mostly activated by Stage 2 mid-training — which is the central finding of the paper's RQ4.

Usage

import json, torch, torch.nn.functional as F
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoImageProcessor
from transformers.models.qwen2_5_vl.modeling_qwen2_5_vl import Qwen2_5_VisionTransformerPretrainedModel
from transformers.models.qwen2_5_vl.configuration_qwen2_5_vl import Qwen2_5_VLVisionConfig

path = snapshot_download("Pixel-Linguist/Pixel-Linguist-II-Pretrain")

config = Qwen2_5_VLVisionConfig(**json.load(open(f"{path}/config.json")))
model = Qwen2_5_VisionTransformerPretrainedModel(config)
model.load_state_dict(load_file(f"{path}/model.safetensors"), strict=True)
model = model.to("cuda", torch.bfloat16).eval()
processor = AutoImageProcessor.from_pretrained(path)


@torch.no_grad()
def encode(images):
    """Embed a list of PIL images (rendered text or natural photos)."""
    inputs = processor(images=images, return_tensors="pt")
    pixel_values = inputs["pixel_values"].to("cuda", torch.bfloat16)
    grid_thw = inputs["image_grid_thw"].to("cuda")

    hidden = model(hidden_states=pixel_values, grid_thw=grid_thw)

    sizes = ((grid_thw[:, 1] // 2) * (grid_thw[:, 2] // 2)).long()
    if sizes.sum() != hidden.shape[0]:
        sizes[-1] += hidden.shape[0] - sizes.sum()
    index = torch.repeat_interleave(torch.arange(len(grid_thw), device="cuda"), sizes)

    pooled = torch.zeros((len(grid_thw), hidden.shape[-1]), dtype=hidden.dtype, device="cuda")
    pooled.index_add_(0, index, hidden)
    pooled = pooled / sizes.unsqueeze(1).to(hidden.dtype).clamp(min=1)
    return F.normalize(pooled, dim=-1)

To embed a string, render it to an image first — that is the whole point of the model. Any renderer works; the paper samples from 393 fonts and 5,000+ DTD textured backgrounds during training so the encoder is robust to layout:

from PIL import Image, ImageDraw

def render(text, size=(448, 64)):
    img = Image.new("RGB", size, "white")
    ImageDraw.Draw(img).text((5, 25), text, fill="black")
    return img

emb = encode([render("a dog runs in the park"),
              render("a puppy is running outside"),
              render("quantum field theory")])
emb @ emb.T   # 0.67 for the paraphrase pair, 0.11 for the unrelated one

Reproducibility notes. transformers>=4.57 loads the image processor as Qwen2VLImageProcessorFast by default and warns that this "may produce slightly different outputs"; pass use_fast=False to AutoImageProcessor.from_pretrained if you need the slow processor. Scores are also sensitive to how text is rendered — the canvas size and font size act as spatial proxies for this model, so the same string rendered at a different width will give a different embedding.

Training

Text corpus	62M multilingual Wikipedia docs, each randomly cropped twice into unsupervised positive pairs
Image corpus	26M natural image-text pairs from LAION-2B
Objective	Symmetric in-batch InfoNCE, embeddings all-gathered across ranks, logit_scale = 1/0.03
Epochs	2 (176M examples seen)
LR / batch	5e-5 / 1024
Infra	64 GPUs, bf16, DeepSpeed ZeRO-2, gradient checkpointing

Text is rendered on the fly with randomised fonts, sizes, backgrounds, brightness and blur, so the model never sees the same visual instantiation of a string twice. This suppresses the pixel-level shortcut learning documented in the paper's RQ3. Natural image-text pairs act as a required regulariser — dropping them collapses document retrieval (RQ2).

Citation

@inproceedings{yuan2026pixel,
  title     = {On the Design Fundamentals of Pixel Text Representation Learning},
  author    = {Yuan, Chaohao and Yuan, Ruifeng and Huang, Zhuoxu and Rong, Yu and
               Cheng, Hong and Chan, Hou Pong and Xiao, Chenghao},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year      = {2026}
}