gemma-4-e2b-ltx-2-5-prompt

Full bf16 fine-tune of google/gemma-4-E2B-it that converts a story into a single highly detailed LTX 2.5 cinematic video prompt — the most emotionally intense, visually compelling scene of the story, rendered as a shot-by-shot prompt (camera, lighting, motion, lens, atmosphere).

Model description

• Base model: google/gemma-4-E2B-it (Gemma 4 E2B IT).
• Fine-tuning method: full bf16 fine-tune (no quantization, no adapter). The base model loads as Gemma4ForConditionalGeneration (multimodal); the vision/audio towers are frozen and only the text decoder (language_model + lm_head) is trained — this is a text-only model despite the multimodal checkpoint.
• Compute dtype: bf16.
• Task: text generation — story → LTX 2.5 prompt.

Training data

Chat-formatted examples (data/dataset.jsonl), one conversation per example: a leading instruction + the story as the user turn, and the target LTX 2.5 prompt as the final assistant turn. The prompt (instruction + story, joined as {instruction}\n\n### CONTEXT\n{story}) is masked to -100, so the loss is computed only on the assistant completion. Examples are pre-tokenized by prepare_dataset.py; the assistant turn is located by its start marker (a special token derived from the generation-prompt diff), which sidesteps TRL's fragile prompt/completion prefix check.

• Train examples: 8958
• Validation examples: 497

Training hyperparameters

Parameter	Value
Epochs	1.0
Max length	4096
Micro batch	16
Gradient accumulation	1
Effective batch	16
Learning rate	0.0001
LR schedule	cosine (~3% warmup)
Weight decay	0.01
Max grad norm	0.3
Optimizer	adamw_torch_fused
Gradient checkpointing	True
Precision	bf16

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "autopostflow/gemma-4-e2b-ltx-2-5-prompt"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
).eval()

instruction = "<your system instruction — You are an expert cinematographer…>"
story = "<the story>"

user_content = f"{instruction}\n\n### CONTEXT\n{story}"
messages = [{"role": "user", "content": user_content}]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
)
inputs = {k: v.to("cuda") for k, v in inputs.items()}

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=4096,
        do_sample=True,
        temperature=0.9,
        top_p=0.95,
        top_k=64,
        repetition_penalty=1.05,
        pad_token_id=tokenizer.pad_token_id,
    )

prompt = tokenizer.decode(
    out[0, inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
).strip()
print(prompt)

The prompt layout (### CONTEXT separator) must match what the model was trained on — keep it consistent at inference.

Intended use

Generate LTX 2.5 video prompts from narrative text. Feed the model's output to an LTX 2.5 video generator as the text prompt.

License

Derivative of google/gemma-4-E2B-it, distributed under the Gemma Terms of Use. See the Gemma license for terms and acceptable use policies.