VincentGOURBIN/fluxforge-director-gemma4-e2b-lora-v1

Fluxforge Director LoRA (Gemma 4 E2B) — v1

A LoRA adapter that turns Gemma 4 E2B into the "AI Director" for Fluxforge Studio: given a short creative brief (plus an optional subject photo identity sheet), it plans a short film as a single JSON document — a title, synopsis, casting, and a shot list using three tools (generate_clip, speak_line, set_casting) that Fluxforge Studio's local rendering pipeline (LTX video, Klein image editing, TTS + LipDub) executes directly.

Why this exists

The base E2B model (quantized 6-bit, the variant the app actually runs) is unreliable at this task out of the box — real-world testing measured roughly a 1/3 failure rate on malformed JSON, missing required fields, and near-total avoidance of the speak_line tool even when available. This adapter was trained via distillation: a stronger teacher model (GLM-5.3-Flash) generated 1000 example plans across a diverse bank of briefs, iteratively curated (both automatic — schema validation, quality-defect detection and fixing — and manual review) into a clean training set, then used to fine-tune this LoRA.

Training

• Base: mlx-community/gemma-4-e2b-it-bf16 (bf16, non-quantized — training precision; the adapter loads correctly onto the quantized 6-bit base the app actually runs at inference time).
• Method: gemma4-cli lora train (gemma-4-swift-mlx), LoRA rank 8, scale 20, 16 adapted layers, response masking (loss on the assistant turn only), learning rate 1e-4, 1 epoch (897 examples).
• Data: 1000 teacher-generated plans (GLM-5.3-Flash), schema-validated and iteratively curated for quality (dialogue-language accuracy, no tool-name leakage into visual prompts, no fabricated dialogue when lip-sync isn't available), split 897 train / 103 valid.
• Result: validation loss 1.79 → 1.00 (-44%), monotonically decreasing, no overfitting signs.

Evaluation (30 held-out briefs, never seen in training)

Schema-validity rate, base E2B vs this adapter vs the GLM-5.3-Flash teacher, same prompts:

Condition	Valid plans
Base E2B (no adapter)	17/30 (57%)
E2B + this adapter	29–30/30 (97–100%)
Teacher (GLM-5.3-Flash)	24–26/30 (80–87%)

The adapter matches or exceeds the teacher's own schema reliability on unseen briefs, and clears the base model's most common failure modes (truncated JSON, missing tool field, missing required focus).

Known limitation: this adapter under-uses the optional transition_in field (a stitched transition clip between shots) relative to the training data's own rate — schema validity and core planning quality are unaffected, but the current release doesn't reliably lean into transitions as a stylistic choice. Not yet resolved as of this release.

Usage

Load on top of the base model with gemma-4-swift-mlx's Gemma4LoRAInference:

gemma4-cli lora generate \
  --model-path <path-to-gemma-4-e2b-it-6bit-or-bf16> \
  --adapter-path <path-to-this-repo> \
  --system "<Fluxforge Director system prompt>" \
  "<brief>"

The exact system/user prompt format this adapter was trained on is generated by Fluxforge Studio's DirectorPromptBuilder (prompt schema version 6) — using this adapter with a different prompt format is not expected to work well.

License

Inherits the base model's Gemma license. This adapter's weights (not the base model) are released under the same terms.