See us on the Leaderboard ⇗

PixelModel v3 🖼️

A neural network where the weights are the image. Same gimmick as v1 and v2: model.png is not a picture of the model, it is the model. Every pixel encodes a weight. At inference the PNG is decoded back into weight matrices, the prompt is embedded, and a coordinate network paints an image at any resolution.

The difference this time is the network. v2 was v1 with wider layers and nothing else (its own card says "that's the only change"). v3 keeps the PNG idea and rebuilds the architecture: learned word embeddings instead of a fixed hash, sine activations instead of tanh, and FiLM conditioning instead of concatenation. It is 919,427 parameters, and on the same eval pipeline it beats both v1 and v2 on FID and on CLIP.

What changed from v1

v1's real limits were three specific things, and v3 replaces each one.

Learned word embeddings. v1 hashed the prompt into a fixed vector with zero learnable parameters, so the text path could never learn what a word means. v3 has a real embedding table over the common COCO caption words (lowercase, whitespace tokenised), mean pooled into the latent.
A SIREN decoder. tanh smears high frequencies. v3 uses sine activations with the proper SIREN initialisation (first layer uniform on plus or minus 1/fan_in, later layers uniform on plus or minus sqrt(6/fan_in)/w0, w0 of 30). Getting that init wrong silently kills training, so train.py prints the per layer activation statistics in the first few steps as a check.
FiLM conditioning. Instead of gluing the latent onto the coordinates, the latent generates a per layer scale and shift that modulate each decoder layer. The FiLM generator is zero initialised, so at step 0 the scale is 1 and the shift is 0 and the SIREN statistics are left intact.

Results

On the same eval pipeline, v3 posts the best FID and the best CLIP of the three.

Metric	v1	v2	v3
Parameters	23,747	~200,000	919,427
FID, published	439.46	not published	see below
CLIP, published	20.02	not published	see below
FID, this pipeline (lower better)	420.75	390.68	383.91
CLIP, this pipeline (higher better)	20.10	20.48	20.73
`model.png`	160x149	scaled	959x959 (1.78 MB)
Native resolution	64	64	128

Read this carefully, because the fair comparison is not v3 against a published number. It is every model re run through the exact same code.

Re scoring v1 here gave FID 420.75, not the published 439.46, and CLIP 20.10, not 20.02. That roughly 19 point FID gap is library and subset drift, not a quality change. Comparing v3 to a published number from a different setup would have overstated the win, which is why the regression is not optional.
v2 never published an FID or CLIP (its card lists both as "NOT DONE"), so it was re scored here from its own weights through the identical pipeline: FID 390.68, CLIP 20.48.
On the same pipeline the ranking is unambiguous. FID: v3 383.91 < v2 390.68 < v1 420.75. CLIP: v3 **20.73** > v2 20.48 > v1 20.10. v3 wins both, at about 39 times v1's parameter count and about 4.6 times v2's.

The thing that got v3 there was matching train and inference scale. v3 renders and is scored on whole frames, so it is trained on whole frames at the same size, rather than on zoomed in crops. Training on crops teaches the network patch statistics that never appear when it paints a full image, and closing that gap is what moved FID from 392 to 384 and CLIP from 19.9 to 20.7 without touching the architecture.

Everything labelled "this pipeline" is n=5000, MS COCO val2014 with a 256 center crop, scored with torchmetrics (FrechetInceptionDistance and CLIPScore with openai/clip-vit-base-patch32), same eval subset and same library versions. The eval images are hash checked to be disjoint from training.

Training

80 epochs on a single RTX 3090, about 40 minutes, batch 32, 128x128 whole frames, cosine learning rate from 2e-4, mixed precision. Training loss goes from 0.0720 to 0.0540 and has flattened by the end.

On scaling up

I also trained a larger variant (about 3.4M parameters, wider and deeper, at 256x256) to see whether the extra capacity helps. It does not. At the base learning rate it would not descend at all, and even after dropping the learning rate its best loss (0.070) never reached base's 0.054 before it destabilised. A deeper SIREN with a large FiLM hypernetwork is simply harder to optimise here, and the extra parameters buy nothing for this task. So the released model is the 919K base. That is the honest outcome, not a hidden one.

Architecture

prompt string
  lowercase / whitespace tokenize -> token ids (up to 20)
  learned embedding table (3923 x 64), mean pool over real tokens   [251,072]
  Linear(64 -> 256) -> sin
  Linear(256 -> 192)                                        = latent z (192)

FiLM generator (zero init, so scale=1 / shift=0 at start):
  z -> Linear(192 -> 2 x 256 per sine layer)                = (scale, shift) x4  [395,264]

for every pixel (x, y), decoded as one batched tensor (no python loop):
  Fourier features of (x, y): [x, y] plus sin/cos over 8 octaves = 34 dims
  SineLayer 0: Linear(34  -> 256) -> FiLM -> sin(30 * .)    (SIREN first layer)
  SineLayer 1: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 2: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  SineLayer 3: Linear(256 -> 256) -> FiLM -> sin(30 * .)
  head: Linear(256 -> 3) -> sigmoid                         = RGB

Because pixels come from coordinates, the parameter count does not depend on resolution: --res 256 runs the same 919,427 weights as --res 64.

block	tensors	parameters
embedding table	`embed.weight`	251,072
text encoder	`text_fc1`, `text_fc2`	65,984
FiLM generator	`film`	395,264
SIREN decoder	`sine_layers.0..3`	206,336
RGB head	`head`	771
total		919,427

Weight encoding

Same scheme as v1, sized up. Each pixel stores one weight at 16 bit precision, mapped linearly from the range [-2, 2]:

R channel is the high byte
G channel is the low byte
B channel is reserved

At 919,427 weights the PNG is a roughly square 959x959 image of 1.78 MB that loads in about 30 ms. Round trip quantisation error is about 3e-5 per weight, and no trained weight fell outside [-2, 2]. model.safetensors holds the identical weights in standard format with the full parameter breakdown in its header.

Usage

# build the train and eval data plus the vocabulary from COCO
python fetch_coco_subset.py --out ../pm-work

# train (writes model.png and config.json every epoch)
python train.py --data ../pm-work/coco_train.npz --vocab ../pm-work/vocab.json \
    --crop 128 --batch-size 32

# inference from model.png (the canonical model)
python main.py "a red double decker bus" --out bus.png

# standalone inference from model.safetensors (needs only that file plus torch)
python convert_to_safetensors.py
python INFERENCE.py "a red double decker bus" --out bus.png

# any resolution from the same weights
python main.py "a beach with palm trees" --res 256

# eval: FID and CLIP
python eval/run_eval.py --arch v3 --work ../pm-work --n 5000

# regression: re score v1 or v2 through the identical pipeline for a fair comparison
python eval/run_eval.py --arch precomputed --work ../pm-work \
    --images-dir v1_render/ --n 5000

main.py (from model.png) and INFERENCE.py (from model.safetensors) produce byte identical output for the same prompt and resolution.

What to expect

This is a 919K parameter coordinate network. It is far more capable than v1's 23K, and it tracks caption conditioned colour and coarse layout better, but it does not draw a sharp bus. The point is the size to quality ratio and the honesty of the PNG as model idea, measured by eval/run_eval.py rather than asserted.

Files

model.png                   THE MODEL (959x959), produced by train.py
model.safetensors           same weights, standard format plus param metadata
config.json                 architecture and parameter count metadata
vocab.json                  word level tokenizer vocabulary, from fetch
main.py                     inference, loads model.png
INFERENCE.py                inference, loads model.safetensors (standalone)
convert_to_safetensors.py   model.png to model.safetensors
train.py                    training (AMP, cosine LR, per epoch PNG write)
model.py                    architecture, tokenizer and PNG weight codec
fetch_coco_subset.py        builds train and eval data plus vocab from COCO
eval/run_eval.py            FID and CLIP Score via torchmetrics

Environment

torch has no wheels for Python 3.14 yet, so use Python 3.11 or 3.12.

uv venv --python 3.11 .venv
# training
uv pip install torch numpy pillow safetensors datasets
# eval (FID and CLIP). torchmetrics CLIPScore breaks on transformers 5.x, so pin
# transformers to 4.49.0 or the metric raises inside torchmetrics.
uv pip install torchmetrics torchvision torch-fidelity scipy "transformers==4.49.0"

Training and eval here ran on a rented RTX 3090, torch 2.13.0+cu130. fetch_coco_subset.py downloads the split before iterating rather than streaming it, because a long streaming loop dies to dropped connections partway through.

Bench Labs. Simple, Reliable, Open sourced.