vlm-data-in-weights

Small neural generators that store a vision-language dataset in their weights and replay it without reading the original files. This is an experiment in synthetic data generation for the VLM backbone of a vision-language-action (VLA) model: how faithfully a QA corpus with images can be kept in compact model weights and regenerated offline.

What the weights contain

The generators were trained to reproduce existing public data, not to invent new data.

Benchmark part (generator/benchmark/): questions, answer options, correct answers and images of the publicly released items of these VLM benchmarks — BLINK, CrossPoint-Bench, EmbSpatial-Bench, ERQA, MMMU (validation), MMMU-Pro (standard and vision) and MSMU; 12,288 items in total. The original datasets' licenses and terms apply to the regenerated content.

Robot part (generator/robot/): camera frames and visible-object-list annotations from 130 LIBERO simulation tasks (4,160 QA records).

License

Our code and the trained weights as our work are released under MIT. The content the weights reproduce is not ours: regenerated benchmark items remain under the original terms of each benchmark (BLINK, MMMU, MMMU-Pro: Apache-2.0; ERQA, EmbSpatial-Bench, CrossPoint-Bench, MSMU: see their pages), LIBERO frames under LIBERO's terms, and the bundled stabilityai/sd-vae-ft-mse codec under MIT.

Architecture

• Text: 16 token-counter MLPs (width 1024, depth 6, ~5.3M parameters each). A position counter with binary/Fourier features is mapped to the 16-bit ID of the token at that position (SmolVLM2 tokenizer), until a learned EOS. Trained with BCE-with-logits/AdamW until the full text stream and EOS were reproduced exactly. These models memorize; they are not general language models.
• Images: 46 latent-counter models (29 benchmark shards, 17 robot shards, up to 512 images each) that map an image index to a 4×64×64 latent of the frozen stabilityai/sd-vae-ft-mse decoder (MIT, included under codec/). Trained in FP32, stored in FP16.
• Robot text fields: one shared counter model.

Usage

python generator/benchmark/generate.py --export generator/benchmark --out OUT --groups <group names>
python generator/robot/generate.py --export generator/robot --out OUT --groups <group names>

A CUDA GPU is required. Group names are listed in generator/benchmark/text_registry.json. The scripts contain a file-access guard that refuses to read the original corpora during generation, so that the output provably comes from the weights.

Training code is in generator_training/code/.