SahilSidhu/2dvideogen-scene-writer

2DVideoGen scene writer (v3-staged)

A t5-small (60M parameters) fine-tuned to turn a natural-language prompt into a scene script — a small symbolic DSL that the 2DVideoGen compositor renders directly to an .mp4.

natural language  ->  this model (t5-small, CPU)  ->  scene-script DSL  ->  compositor  ->  .mp4

No GPU is used anywhere in that path. Training took 548.5 s on CPU (8 threads, 2250 steps, 6 epochs); the two earlier checkpoints in the same family trained in 530 s and 531.7 s.

A ready-to-run Gradio demo of this whole pipeline lives in hf_space/ in the GitHub repository (prompt box, the generated DSL, and the rendered video side by side, CPU only).

Example

Prompt:

two friends meet in the park, one waves, then they kick a ball around

Output — the whole scene is this one line of DSL, shown wrapped:

bg park dur 91 | cast ana blue 96 100 ; bo amber 92 90 |
prop ball 58 100 ; tree 69 110 |
tl 1 26 bo walk 92 ; 8 28 ana wave ; 28 47 bo kick ball ana ; 57 79 bo walk 92

Usage

import os
os.environ["USE_TF"] = "0"   # transformers 4.57 otherwise tries a Keras 3 backend and dies
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tok = AutoTokenizer.from_pretrained("SahilSidhu/2dvideogen-scene-writer")
model = AutoModelForSeq2SeqLM.from_pretrained("SahilSidhu/2dvideogen-scene-writer")

prompt = "two friends meet in the park, one waves, then they kick a ball around"
enc = tok("prompt2scene: " + prompt, return_tensors="pt", truncation=True, max_length=160)
out = model.generate(**enc, max_length=256, do_sample=True, num_beams=1, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

The input must carry the task prefix prompt2scene: . Parsing, validation, repair and rendering live in the GitHub repo (model/scene_grammar.py, model/scene_infer.py, scenescript.py).

Sample. Do not use beam search.

This is the single most important thing to know about this checkpoint, and it was the most surprising finding of the project. Under beam search the generated casts pile on top of one another and the model looks as though it never learned staging. It did. Beam search was reporting the mode of a wide positional distribution.

Same weights, one decoding argument changed, 30 scenes:

decoding	staging score	distinct x positions across 30 scenes
num_beams=4	0.188	3
do_sample=True, top_p=0.95, T=1.0	0.862	55

Measured again over the model's own 150-prompt validation set, the staging score goes 0.182 (beam4) -> 0.875 (sampled), and the fraction of scenes clearing the weaker hand-written reference clip (staging 0.825) goes 0.0% -> 74.3%.

Sampling is not free in distribution: actions-per-character falls 85.7% -> 72.5%, stated-layout obedience 45.8% -> 25.3%, and valid, no repair 80.7% -> 60.7%. Out of distribution it is not a trade at all — every-stated-field goes up, 70.8% -> 75.0%, while staging goes 0.099 -> 0.804.

Also use the unique-colour logits processor (model/scene_constrain.py in the repo). Colour uniqueness across the cast is a global constraint an autoregressive decoder cannot see; the previous checkpoint emitted 71/150 duplicate-colour violations in distribution. With the processor this checkpoint emits 0/150 in distribution and 0/24 out of distribution.

Measured rates

v3-staged, beam4 with the colour processor unless the sampled figure says otherwise. In distribution = 150 held-out validation prompts. Out of distribution = 24 hand-written prompts.

	in distribution	out of distribution
parse	98.0% (99.3% sampled)	100.0%
render, after repair	98.0% (99.3% sampled)	100.0%
valid, zero problems, no repair	80.7% (60.7% sampled)	70.8% (41.7% sampled)
duplicate-colour problems	0	0
cast count	100.0%	95.8%
background	100.0%	100.0%
actions	85.7% (72.5% sampled)	83.3% (87.5% sampled)
every stated field at once	85.7% (layout excluded, like-for-like with v2)	70.8% beam4, 75.0% sampled
staging	0.182 -> 0.875 sampled	0.099 -> 0.804 sampled

Final eval_loss 1.0042 — higher than both earlier checkpoints (v2 0.9470, v1 0.9240), on a higher-entropy target. Loss is not the metric here.

Two earlier checkpoints exist and are documented in the repo's model/MODELS.md. The headline in-distribution figure of 86.0% every-stated-field belongs to v2-deconfounded; v3-staged scores 85.7% on the like-for-like comparison, having paid that fraction of a point for +8.3 points out of distribution and for staging.

Training

• Base: t5-small, 60M params, ~242 MB fp32.
• Data: 6000 synthetic prompt/DSL pairs plus 700 validation, generated by model/scene_synth.py with --deconfound --stage --max-cast 8. Train and validation prompts are disjoint.
• Command: --epochs 6 --batch 16 --lr 3e-4 --max-in 160 --max-out 256 --val-cap 200, CPU only.
• --max-out 256 is not optional: v3 targets have mean 113.9 tokens and max 252; at the previous 176 cap, 11.28% of training targets would be silently truncated mid-scene.

Two deliberate properties of the data are why this checkpoint exists:

• Deconfounding. In the first dataset, clause count equalled cast size in 5681 of 5688 examples (99.9%). The model appeared unable to count a cast; a controlled probe showed it was reading clause count, not the numeral, 15/15. Deconfounding cut the generalisation gap from 65.7 to 23.5 points.
• Joint position sampling (--stage), which is what makes the staging distribution above exist at all.

Limits

• Cast size saturates at the trained ceiling. Trained on casts of 1-8, the model reads the numeral correctly 15/15 for sizes 6-8 and 14/15 for 1-5. Asked for nine it emits a cast of 6, every time, at every clause count — a clean ceiling, not random failure. This is the known seq2seq extrapolation limit, relocated by exactly the amount the training range was widened, not removed. Training on 1-12 would move it to 13.
• Spatial language is not obeyed above chance. Supervising "on the left", "facing each other" and similar gave about +2.7 points over a mismatched-pair control in distribution, and went negative under sampling. This was a stated prediction, and it was falsified.
• parse 98.0% in distribution is a length cap, not a format failure: 7-8 character casts run past max_length=256. Raise it for large casts.
• The DSL vocabulary is closed (ten colours, a fixed action set, three backgrounds, a fixed prop list). Sampling occasionally emits an out-of-vocabulary token — e.g. yellow, which is not one of the ten colours — that the strict parser correctly rejects. Grammar-constrained decoding beyond the colour processor is the obvious unfinished work.
• Stick figures, not styled animation. Styled anime output was ruled out by measurement at this hardware budget; see the repo's ATTEMPTS.md.
• No user study, and no baseline comparison against another system. Both are real gaps.

Provenance

Every number here comes from ATTEMPTS.md (Attempts 22 and 23) or model/MODELS.md in the GitHub repository, where the failures are logged alongside the successes.