Movetoy

A tiny text-to-video model trained from random weights on one RTX 4070 by Claude and Codex (GPT). It reads any text and turns it into a 2-second, 16-frame, 32×32 clip of coloured shapes doing things. It is deliberately small and often wrong, but it really does respond to what you write.

movetoy.pt (51,029,040 parameters). Code to load it is in tiny_video/.

Load it yourself

pip install torch numpy imageio[ffmpeg] huggingface_hub

import torch, imageio
from huggingface_hub import hf_hub_download, snapshot_download
import sys; sys.path.insert(0, snapshot_download("Kompaktion/Movetoy", allow_patterns=["tiny_video/*"]))
from tiny_video.model import load_checkpoint, sample, tokenizer_for, to_uint8

model, _ = load_checkpoint(hf_hub_download("Kompaktion/Movetoy", "movetoy.pt"), "cpu")
tokens = tokenizer_for(model.cfg)(["a red circle and blue circle merge into a purple circle"])
video = sample(model, tokens, steps=24, cfg_scale=3.0, generator=torch.Generator().manual_seed(0))
frames = to_uint8(video)[0].numpy().repeat(8, 1).repeat(8, 2)          # 16 frames, 32x32 -> 256x256
imageio.mimwrite("movetoy.mp4", list(frames), fps=8, macro_block_size=1)

Runs on CPU (about 0.8 s per sampling step with 2 threads) or any GPU. Try it in the browser, or call the free API (gradio_client / HTTP, no key, up to 15 s videos): Movetoy Space.

What it was trained on

A procedural world (tiny_video/world.py) generated on the fly in RAM. Nothing was downloaded and no pretrained weights were used:

• 12 colours (with synonyms like crimson, violet, grey), 9 shapes (circle/ball/dot, square/box, triangle, star, ring/donut, cross, heart, diamond, hexagon), 5 sizes, 7 backgrounds (black, white, gray, night, sky, sunset, grass).
• 16 single-object events: moves in 8 directions (with many verbs), falls, rises, jumps, bounces, zigzags, circles, grows, shrinks, spins, stays still, fades in/out, blinks, changes colour, teleports.
• 7 two-object events: merge with colour mixing (red+blue→purple, red+yellow→orange, blue+yellow→green, …), split, collide, chase, orbit, swap places, and two independent actions.
• Many caption templates, random articles, word order, case and punctuation.

Architecture

Pixel-space video diffusion transformer with rectified flow: 4×4 patches, 8 blocks (dim 384, 6 heads) with joint space-time self-attention, 3D rotary positions (t, y, x), cross-attention to the caption, adaLN-Zero conditioning, and classifier-free guidance. The text encoder is a from-scratch 4-layer transformer over [BOS] bytes [SEP] hash(word)…, where hash(word) = crc32(casefold(word)) mod 16384.

Training (≈112 min on the 4070, 23,874 optimizer steps, batch 16)

phase	steps	time	what changed and why
1	7,874	36 min	raw-byte text only, from random init. The model learned generic scenes but ignored the caption: with right, wrong and no caption, the error was the same (0.0284 / 0.0284 / 0.0282). Stopped early.
2	10,000	48 min	added hashed-word tokens (byte rows and visual weights carried over); training biased toward high noise; unmentioned background = black; 8% of clips were your exact merge example. The caption started to matter, but the model over-learned merging (every two-object scene became a merge).
3	6,000	28 min	rebalanced: merges 30%→13% of clips, featured example 8%→2%; lr 1e-4. Exported (raw weights).

The right caption's advantage over a shuffled one, measured at high noise (t = 0.9 / 0.98), went from ~0 in phase 1 to 56% / 66% by the end. verification.json reconstructs the whole lineage from the seeded random init (phase 1 → 2 → 3 hashes) and shows flow loss on 16 fresh clips falling from 1.568 (random weights) to 0.0237.

How well it works

Fixed suite tiny_video/world_eval.py: 10 prompts × 4 seeds, 64 sampler steps, guidance 3. The pixel checks are initial/final object counts, colours, final shape, and the action itself. As sanity checks, rendered references pass 100%, and blank, noise, reversed, and static controls pass 0%.

prompt	all checks pass
a red circle and blue circle merge into a purple circle (your example)	3/4
Two balls, one red and one blue, come together and become one purple ball.	2/4
A blue circle fuses with a red circle to form a single violet circle.	2/4
a yellow circle and red circle merge into an orange circle	1/4
A purple ball floats upwards.	1/4
A small blue square grows bigger.	0/4 (it does grow in 4/4, but into a blob, not a square)
A red circle changes color and becomes blue.	0/4
A little green ball glides from left to right.	0/4
A red circle and a blue circle collide and bounce apart.	0/4
A purple circle splits into a red circle and a blue circle.	0/4
overall	9/40 (22.5%)

Plain words: merging and colour mixing work well, colours are almost always right, and backgrounds follow the text. Size changes and simple motions are hit-or-miss (direction is often wrong), and collisions/splits don't work. Shapes are usually blobby at 32×32. Everything is clearly neural-network output: no frames are drawn by the renderer at generation time, and there's no interpolation.

Fixed-seed demo (outputs/demo-movetoy/, one seed per prompt): 40% of event checks, including the exact merge prompt.