JoshHughes/sana-video2-firstframe-lora

SANA-Video 2.0 5B 4-step: First-Frame LoRA (f3)

A rank-32 LoRA for NVIDIA's SANA-Video 2.0 5B 720p 4-step model that teaches it two picture-conditioned tasks it was not trained for: picture to video (the clip starts on your picture and stays on it) and first + last frame (the clip starts on one picture and lands on another). NVIDIA released the 4-step model as a text-to-video research preview, "not intended for first-frame image conditioning". The transformer does take a timestep per latent frame, though, so a picture can be held in place ("pinned") while the rest of the clip is generated. Without training, the student often drifts off a pinned picture, and with two pinned pictures it tends to hold still and then snap to the end. f3 was trained only on clips that SANA-Video 2.0's own 4-step model generated and that were then verified (self-distillation: the model is its own teacher). No other video model and no real video were used. On 100 held-out pictures it stays on the subject more often (CLIP), and on first + last frame it lands closer to the target with 42 % more motion and a much smaller end snap. All of these numbers are reproducible with the scripts and inputs in eval/.

Author: Josh Hughes · Licence: Apache-2.0 · Base model: Efficient-Large-Model/SANA-Video_2.0_5B_720p_4step (Apache-2.0)

Example outputs

Every example has three rows: the picture(s), the same model without the LoRA, and with f3. Both clips use the same prompt, seed and picture(s), 832x480 (or 480x832), 81 frames at 16 fps. They are the best-scoring examples from the held-out test set. The full test set and the per-clip numbers are in eval/.

Task	Clip	Subject
First frame	ff_bram_2.mp4	A box-headed robot. Without the LoRA the box lifts off and a human face appears. With f3 it stays the robot.
First frame	ff_atlas_5.mp4	A paper-craft whale over paper clouds. Without the LoRA it grows and turns into a different paper shape.
First frame	ff_mochi_3.mp4	A plush character in a toy room
First frame	ff_glim_2.mp4	A pixel-art jellyfish wizard
First frame	ff_sorrel_2.mp4	A stop-motion fox farmer
First + last frame	flf_atlas_1.mp4	A paper-craft whale lifts its tail fin
First + last frame	flf_bram_1.mp4	A voxel robot walks and raises a hand
First + last frame	flf_glim_1.mp4	A jellyfish wizard in a cave
First + last frame	flf_ines_1.mp4	A hiker on a snowy ridge at sunrise

The first + last examples differ less visibly than the averages below. The test-set numbers are the better guide there.

Model files

File	What it is
sana-video2-5b-4step-firstframe-f3.safetensors	The LoRA. 322 tensors, float32, 128 MB. sha256 b5037ad2e446196e14c814f91eaec5e32cba720480280fab5c01851aafacdab2
eval/	Loader + sampler (sana_video2_lora.py), metric scripts, the 100 test pictures and 100 last pictures, generation jobs, per-clip results
examples/	The side-by-side clips above
p0pc0rn/	The same weights as an fp16 pack for the p0pc0rn on-device WebGPU engine (alpha / rank folded into B). Not needed for PyTorch

Key names are the official module names. Every key is <module>.lora_A.weight [32, in] or <module>.lora_B.weight [out, 32], where <module> is the module's name inside NVIDIA's SanaVideo2_5B (blocks.0.attn.qkv, ..., blocks.31.cross_attn.proj, t_block.1), which is also its key in the released checkpoint minus .weight. There is no prefix and no renaming. NVIDIA's own RL adapter uses the same module names. Metadata: rank 32, alpha 32, rule W' = W + (alpha/rank) * B @ A, base Efficient-Large-Model/SANA-Video_2.0_5B_720p_4step, step 1000. See Key-name check for how this was verified.

Model details

• Base: SANA-Video 2.0 5B 720p 4-step (4.47 B parameters, SanaVideo2_5B, DMD student, sampler fastvideo_dmd_4step / sana_shift6_dpm, CFG 1)
• Adapter: LoRA rank 32, alpha 32 (scale 1.0) on 161 linear layers, 32.0 M parameters:
  • attn.qkv, attn.output_gate, attn.proj, cross_attn.q_linear and cross_attn.proj in all 32 blocks;
  • plus t_block.1, the shared AdaLN projection of the timestep embedding. A pinned frame runs at t = 0, so this layer carries the "this frame is given" signal.
• Tasks: first frame → video; first + last frame → video. Text prompt as usual.
• Trained at: 832x480 and 480x832, 81 frames (11 latent slots), 16 fps
• Inference: the unchanged 4-step sampler (4 transformer calls), no CFG, prompt suffix motion score: 20. (NVIDIA's default motion score)
• Precision: the LoRA is stored in fp32. We ran it on top of the bf16 transformer.

Intended use and out of scope

Intended:

• animating a picture with the fast 4-step model;
• keyframed clips with a known start and end picture;
• research on picture conditioning of distilled video models.

Out of scope:

• Identity verification or depicting real people. The training and test pictures are synthetic characters.
• Ends the motion cannot reach. For example a different camera angle, a new object, or a large change in the scene.
• Resolutions above 1280x736. The base model starts duplicating content there.
• Middle keyframes (pinning a frame in the middle of the clip). This was not a training task, and the clip lurches around the pinned frame.

How it works

Pins. The LTX-2.3 VAE packs 81 frames into 11 latent slots: slot 0 = frame 0, then slot k = frames 8k-7 ... 8k. To condition on a picture:

1. Encode it as a single still.
2. Write its latent into slot 0, for first frame, or into slots 0 and 10, for first + last.
3. Run those slots at timestep 0, while the free slots run at 1000·σ. The transformer's frame-aware AdaLN takes one timestep per latent frame, which is the mechanism NVIDIA's TI2V path uses.
4. Re-write the pinned slots after every one of the 4 stages.

Nothing in the model architecture changes. This is what sana_video2_lora.sample(..., pins=...) does.

Why the base model needs help. Three things we measured on the base model:

• Picture to video. The 4-step student was distilled for text-to-video only. With a pinned first frame it often drifts: faces change, objects transform, and the art style slides toward photoreal.
• First + last frame, stage 0. With two clean pinned slots it cannot connect, so the model decides in stage 0 (σ ≈ 1) to move far less in between. Stages 1-3 barely change that.
• First + last frame, slot 10. The last picture, packed as a still, is far from any latent that a real 8-frame chunk produces in slot 10. So the end arrives as a jump into the last slot (frames 73-80).

What f3 learns.

• From verified clips the model itself produced, it learns to hold the pinned first frame.
• From verified clips rendered with a late end pin, it learns to spread the motion over the clip and land on a still-packed last picture, with the end pinned the normal way.

Usage

The loader works with NVIDIA's own model code (NVlabs/Sana, main branch) and the released checkpoint. eval/sana_video2_lora.py is self-contained: model build, LoRA, pinned 4-step sampler, text encoder and VAE.

import torch, sys
sys.path.insert(0, "eval")                       # sana_video2_lora.py from this repo
import sana_video2_lora as L                    # needs NVlabs/Sana importable (the `diffusion` package)

dev = "cuda"
model = L.load_model("SANA_Video_2.0_5B_720p_4step.pth", device=dev)            # official checkpoint, bf16
L.apply_lora(model, "sana-video2-5b-4step-firstframe-f3.safetensors")          # attach by official module name (merge=True to fold in)
te, vae = L.TextEncoder("gemma-2-2b-it", device=dev), L.Vae("LTX-2.3-Diffusers", device=dev)

H, W = 480, 832
cond, mask = te("A paper-craft whale floats in a paper ocean ... The whale lifts its tail fin, then settles.")   # describe the picture + the motion
first = vae.encode(L.read_picture("first.png", W, H)[None])
last = vae.encode(L.read_picture("last.png", W, H)[None])                     # optional: first + last frame
lat = L.sample(model, cond, mask, L.latent_shape(H, W, 81), H, W, seed=1, pins={0: first, -1: last})
L.write_mp4(vae.decode(lat), "clip.mp4")

Command line: python eval/sana_video2_lora.py i2v --ckpt ... --text_encoder ... --vae ... --lora sana-video2-5b-4step-firstframe-f3.safetensors --picture first.png [--last last.png] --prompt "..." --seed 1 --out clip.mp4.

The attach path was checked to give bit-identical latents to the training code (see Key-name check). Merging (merge=True) adds the delta into the bf16 weights, and that rounding is not negligible: on the tiny check, the merged latent differs from the attached one by about 14 % of the LoRA's own effect. Attach for exact results. All our numbers use the attached LoRA at scale 1.0.

Recommended settings

• LoRA scale 1.0. It was trained and tested at 1.0.
• Prompt: one paragraph describing the picture (subject, look, setting, lighting, framing), then what happens. On the base model alone, this prompt form keeps the clip on the subject much more often than a motion-only prompt (65 → 76 of 100 in our test). f3 was trained with it.
• Size: 832x480 or 480x832, 81 frames, 16 fps. 1280x736 is the model's native bucket and also works. Do not go higher.
• First + last: the end picture should be a state the motion can reach, with the same framing, the same subject and the same light. We made ours by editing the first picture with FLUX.2 [klein] 4B: "Show this exact scene a few seconds later, at the last moment of this motion: ... Change only the pose and position that the motion causes."
• Keep the official sampler: the 4 sigmas 0.9998 / 0.9472 / 0.8569 / 0.6664, no CFG. Five steps were worse than four.

Training

The idea: the model is its own teacher

Every training clip was generated by SANA-Video 2.0's own 4-step model, with the picture(s) pinned exactly as at inference, and then verified. Only verified clips were kept. The LoRA learns to reproduce the model's best behaviour with the same inputs that produced it, on the model's own latent trajectories. No clip from any other video model (including the original multi-step SANA-Video 2.0 release) and no real video was used.

The teacher → trainee chain:

Round	Trainer (the model that makes the clips)	Attempts	Verified clips	Trainee
1	base 4-step model, first frame pinned	2,356 (4 per picture, 589 pictures)	522	r1 (not shipped)
2	base + r1, first frame pinned	1,704 (4 per picture, 426 new pictures)	851 from rounds 1 + 2, at most one per picture (544 made by the base model, 307 by base + r1)	r2, trained from the base model
3	base + r2, first frame pinned + end picture pinned late	1,895	111 first + last clips used	f3, trained from r2, with r2's first-frame clips mixed back in

Late end pin (round 3). The end picture enters at stage 1 instead of stage 0. Stage 0 then lays out the motion freely, and the end still gets reached. These clips have no hold, but this rendering mode makes mid-clip pops of its own, which is why it is not used as the inference recipe. The verified ones were used as targets with the end pinned at every stage, the normal hard pin. The LoRA learns to move like the late-pin clips under the ordinary pinning that any app uses.

Pictures. All training pictures were generated with FLUX.2 [klein] 4B (Apache-2.0) from prompts, and are the lab's own invented characters and scenes. The end pictures were FLUX.2 [klein] 4B edits of the first picture. No photographs of real people and no user content were used. Training pictures and test pictures do not overlap.

Objective

One training sample = one stage k of one recorded clip:

x_k     = the clip's recorded input at stage k (its own noise path; one input -> one clip, so no averaging and no blur)
x_k[0]  = the clip's own final slot-0 latent              (clean pin, timestep 0)
x_k[10] = the end picture's still latent, for first+last   (clean pin, timestep 0)
t       = 1000 * sigma_k on free slots, 0 on pinned slots
x0_hat  = x_k - sigma_k * v_theta(x_k, t, prompt)
loss    = mean squared error(x0_hat, clip's final latent) over the free slots (1..9, or 1..10 for first frame only)
k ~ {0, 1, 2, 3} with weights 1 : 1 : 1 : 1.5

Hyperparameters

	Stage A: r2	Stage B: f3 (shipped)
Starts from	base model	r2 at step 1,000
Data	851 first-frame clips	864 clips: 753 first-frame + 111 first + last
Sampling	uniform	217 clips drawn 3x as often (see the note below); in 85 % of the draws of a first + last clip the end is pinned
Steps	1,000 (of a 2,000-step run; later checkpoints tested worse)	1,000
Learning rate	5e-5, linear warm-up 50 steps, then constant	3e-5, same schedule
Optimizer	AdamW β (0.9, 0.999), eps 1e-8, weight decay 0, gradient clip 1.0	same
Batch	1 (one stage of one clip)	1
LoRA	rank 32, alpha 32, A: Kaiming-uniform, B: zeros, fp32	continued
Base	frozen, bf16, gradient checkpointing	same
Seed	1234	1234
Hardware	a single workstation GPU setup (32 GB), PyTorch, ~3.5 s/step, peak 12.4 GB	same, ~1 h

Sampling note (what the shipped checkpoint was actually trained with). The 3x weight was meant for the first + last set. Two data folders reused the same clip ids, so 106 first-frame clips also received it, and the other half of the verified first + last clips were never loaded. The effective mix was 111 first + last clips at 3x, 106 first-frame clips at 3x, and 647 first-frame clips at 1x.

Evaluation

Protocol.

• Test set: 100 held-out pictures. 65 show 13 invented characters (5 each) and 35 are scenes with no main character. Many looks: photo, 3D, voxel, pixel art, watercolor, claymation, paper-craft, felt, cel. None were seen in training.
• Arms: the same model with and without the LoRA, the same prompt, seed and picture(s), 832x480 or 480x832, 81 frames.
• Code: the scripts, pictures, prompts and seeds are all in eval/.

Metrics (public models only):

• First frame: CLIP ViT-L/14 image similarity between the input picture and frames 40 and 80. A clip stays on the subject when both are ≥ the threshold. The table also reports the mean absolute frame-to-frame change ("motion"), so that freezing does not count as success.
• First + last frame: PSNR and SSIM of frame 80 against the target last picture, the median motion over frames 1-72, and the end rush. The end rush is the motion into the last latent slot (frames 73-80) divided by the motion over frames 60-72. 1 means an even pace.

First frame (100 pictures)

	Without LoRA	f3
Stays on the subject, CLIP ≥ 0.85 at frames 40 and 80	76 / 100	84 / 100
Same, threshold 0.80 / 0.90	88 / 44	92 / 51
Characters only (65), ≥ 0.85	44 / 65	51 / 65
Scenes only (35), ≥ 0.85	32 / 35	33 / 35
Mean CLIP similarity, frame 40 / frame 80	0.914 / 0.882	0.919 / 0.892
Paired: frame-80 similarity higher with f3		62 of 100 clips (sign test p = 0.021)
Motion, median (0-255 grey levels per frame)	2.55	2.35
Stays on the subject among clips that move (motion ≥ 1.0)	71 / 95	77 / 93

The prompt alone matters. The same base model with a motion-only prompt stays on the subject in 65 / 100 at 0.85. Describing the picture lifts it to 76, and f3 to 84. f3 moves slightly less than the base model: the median per-clip motion ratio is 0.97, and 62 of 100 clips move less (p = 0.021). The gain is mostly on characters.

First + last frame (100 pictures, end = an edited version of the first picture)

	Without LoRA	f3
End match, PSNR of frame 80 vs the target	26.53 dB	27.20 dB (higher in 93 of 100 clips, p = 2.7e-20)
End match, SSIM	0.840	0.844 (higher in 69 of 100, p = 1.8e-4)
Median motion over frames 1-72	0.663	0.941 (+42 %; higher in 91 of 100, p = 3.3e-18)
End rush, median	2.53x	1.46x (lower in 100 of 100)
Clips with an end rush above 2x	73 / 100	24 / 100
Share of the motion in the first half	54 %	53 %

f3 removes most of the "hold still, then snap" behaviour. It moves more through the clip, the jump into the last slot shrinks, and the end still lands closer to the target.

Where the LoRA changes the model

eval/lora_weight_map.py gives the relative weight change per layer, ‖(α/r)·B·A‖ / ‖W‖ against the released checkpoint. Here it is summed per group of 8 blocks:

	Blocks 0-7	Blocks 8-15	Blocks 16-23	Blocks 24-31	t_block.1
Σ ‖ΔW‖/‖W‖ × 1000	51.7	57.4	121.4	286.3	3.3

• Where: 55 % of the change sits in the last 8 blocks. The top blocks are 29, 31, 30, 28, 26 and 27. Blocks 27 and 31 are two of the eight softmax-attention "anchor" blocks, the only places where frames attend to each other token by token.
• Which layers: attn.output_gate changes most (123.7), then cross_attn.proj (110.7), attn.qkv (106.8), attn.proj (94.1) and cross_attn.q_linear (81.5).
• Per-layer values: eval/results/weight_map_f3.csv.
• Use: a smaller LoRA aimed at blocks 24-31 is a natural next experiment. We have not tried it.

Limitations

• Not a fix for everything the 4-step preview does. Hands, crowded scenes and fast actions can still break. Floating and flying motions often freeze.
• The camera follows its own habits. In our picture-start clips the model tends to a slow leftward pan and push-in, and it rarely obeys "pan right" or "tilt up". f3 was not trained to change that.
• First + last needs a reachable end. If the last picture needs a big change, the model still runs short of motion, and the change can arrive late or as a morph.
• Slightly less motion on plain picture-to-video (see above).
• The pinned last frame is a still. Its last 8-frame chunk can still show a small seam pop at frame 73. The model then decodes slot 10 from a single-image latent.
• Synthetic training pictures only. Tested on invented characters, not real people. The CLIP metric measures picture similarity, not identity.
• One LoRA strength (1.0) was tested.

What we tried that didn't work

These are negative results from the same lab, stated without numbers. Our internal scoring for these runs is not part of this release.

• Training on clips from the original multi-step SANA-Video 2.0 model instead of the student's own. Fresh-noised targets blurred the output, and ODE-paired targets gave more motion but worse faces. Neither beat the base model, and no such clip is in f3.
• More rounds of self-training plateau. A third round on hard face pictures did not lift the result over r2. Neither did a 4x face-weighted loss, cleaner data with stricter filters, or longer training: r2's 1,500- and 2,000-step checkpoints were worse than 1,000.
• First + last, a last-slot loss. We trained the model to "unpack" the still in slot 10 toward the real 8-frame latent. It removed the hold but moved the change into one big step just before the end, and the end match got worse. Dropping that loss (an ablation) behaved the same.
• Decoding the model's own predicted slot 10 instead of the pinned still: more pop, worse end match.
• Soft pins, released after stage k: they hit the target less and lurch more.
• A noised view of the end picture at stage 0 ("end hint"): even heavily noised, it brings the hold straight back.
• Pin boost, scaling the pinned frames' keys in the 24 linear-attention blocks: no gain. The global summaries already carry the pin.
• Five sampling steps instead of four (an inserted or repeated sigma): worse.
• Adding more late-pin data, or camera-move clips with exact crops as end pictures: more motion but more spring-back ("rubber-banding"), and a faster rush into the end on camera moves.
• Camera moves from inverted clips (synthetic camera motion run backward through a flow ODE to get noise/target pairs): broke the hold but taught rubber-banding. The inversion is only approximate, so the pairs do not match.

Key-name check

• Static (eval/verify_lora_keys.py, no GPU): all 161 modules are nn.Linear weights in the released checkpoint with matching shapes. There are no unexpected keys. Result: eval/results/lora_keys_f3.json.
• Numeric (CPU, bf16, a tiny 4-stage pinned run, latent 128x3x4x7 with first + last pins): eval/sana_video2_lora.py loaded the official .pth and attached this file by module name. The training code loaded its own bf16 copy of the checkpoint and its own LoRA wrapper. The two final latents were bit-identical (max |Δ| = 0). Results: eval/results/lora_compat_check.json.

Licence

• This LoRA, the scripts and the examples: Apache License 2.0 (LICENSE), © 2026 Josh Hughes.
• The base model, SANA-Video 2.0 5B 4-step by NVIDIA: Apache-2.0. The LoRA only works together with it.
• What running the base model needs (not included here):
  • the gemma-2-2b-it text encoder, Gemma Terms of Use;
  • the LTX-2.3 video VAE, LTX-2 Community License Agreement, with its use restrictions and the revenue threshold for commercial entities.
  • The LoRA lives in that VAE's latent space but contains no VAE weights.
• Pictures: made with FLUX.2 [klein] 4B, Apache-2.0.
• Evaluation only: CLIP ViT-L/14, MIT.

Full credits and licence notes: NOTICE.md.

Citation

@misc{hughes2026sanavideo2firstframe,
  title  = {SANA-Video 2.0 5B 4-step First-Frame LoRA (f3): self-distilled picture and first + last frame conditioning},
  author = {Hughes, Josh},
  year   = {2026},
  note   = {LoRA adapter for Efficient-Large-Model/SANA-Video_2.0_5B_720p_4step}
}

Please also cite SANA-Video as NVIDIA asks on the SANA-Video 2.0 project page.

Acknowledgments

• SANA-Video 2.0: NVIDIA, the base model and its open code.
• FLUX.2 [klein] 4B: Black Forest Labs, all training and test pictures.
• LTX-2.3 VAE: Lightricks.
• gemma-2-2b-it: Google.
• CLIP: OpenAI, evaluation.
• Built for p0pc0rn, an on-device app that runs this LoRA on a phone. See p0pc0rn/.