Varun95/t2v-finetune-lora

LoRA fine-tune of Wan-AI/Wan2.1-T2V-1.3B-Diffusers for text-to-video generation

---

📖 Overview

Varun95/t2v-finetune-lora is a Low-Rank Adaptation (LoRA) fine-tune of the Wan-AI/Wan2.1-T2V-1.3B-Diffusers text-to-video diffusion model. It is trained to improve prompt adherence, cinematic composition, and temporal coherence for short video clips, while keeping the adapter lightweight (a few hundred MB) so it can be swapped on top of the base model at inference time without duplicating the full backbone weights.

The adapter is designed to be served on commodity GPUs (e.g. a Colab T4 / 16 GB VRAM) using diffusers' CPU-offloading and VAE-slicing memory tricks, and ships with a complete FastAPI + Cloudflare Tunnel serving recipe plus a Next.js polling-based frontend so you can deploy an end-to-end text-to-video product in under an hour.

⚠️ Inference status: set to false in the model card because video generation is VRAM-heavy (≈ 8–10 GB for 480×832 @ 49 frames) and exceeds the free HF Inference API budget. Use the local / Colab recipes below.

---

🏗️ System Architecture

The full system is a polling-based client/server pipeline. The Next.js frontend never holds a long-lived connection open during generation; instead it posts a prompt, gets a job_id, and polls /status every 5 s until the clip is ready, then streams the MP4 back through the Cloudflare Tunnel.

_{Animated when viewed in a browser — data-flow packets travel the tunnel, the denoising frame strip clears, and the progress bar fills.}

Flow legend

#	Direction	Message	Purpose
1	Client → Server	POST /generate {prompt}	Submit generation request
2	Server → Client	200 { job_id }	Immediate acknowledgement
3	Client → Server	GET /status/{ job_id } (every 5 s)	Poll for progress
4	Server → Client	{ status: "processing" }	Still rendering
5	Server → Client	{ status: "completed", video_url }	Done — ready to fetch
6	Client → Server	GET /videos/{ job_id }.mp4	Stream the rendered video

The Cloudflare Tunnel (cloudflared) maps a public https://*.trycloudflare.com URL to the Colab FastAPI port (8000). Closing the Colab tab tears down the tunnel.

---

🧠 Model Details

Field	Value
Model type	Diffusion-based text-to-video (T2V) LoRA adapter
Base model	Wan-AI/Wan2.1-T2V-1.3B-Diffusers (1.3B params)
Adapter type	LoRA (rank-64, alpha-128, all attention/FFN projections)
Precision	torch.float16 (fp16) for inference, bf16 recommended for training
Output	Short MP4 video clips (default 480×832, 49 frames @ 8 fps ≈ 6 s)
Library	🤗 diffusers ≥ 0.31, transformers, accelerate
License	MIT
Developed by	Varun95

Model Sources

Repository	Varun95/t2v-finetune-lora
Base model	Wan-AI/Wan2.1-T2V-1.3B-Diffusers
Diffusers docs	WanPipeline
Original Wan 2.1 paper / repo	Wan-AI/Wan2.1

Inference Pipeline

Internally the model is a standard latent diffusion T2V pipeline: the prompt is encoded to embeddings, random latents z_T are denoised over num_inference_steps by the Wan transformer (with this LoRA adapter injected), and the final latents z_0 are decoded by the VAE into a sequence of frames that are muxed into an MP4.

---

✅ Intended Uses & Out-of-Scope

Intended uses

• Generating short (≤ ~6 s) cinematic video clips from natural-language prompts.
• Creative content generation, storyboarding, and rapid concept visualization.
• Research on text-to-video diffusion, LoRA fine-tuning, and adapter evaluation.
• Prototyping full-stack T2V products on free-tier GPUs (Google Colab T4).

Out-of-scope uses

• Generating photorealistic depictions of real, identifiable individuals without consent.
• Creating deceptive, defamatory, or non-consensual explicit content.
• High-stakes decision-making (medical, legal, surveillance, law enforcement).
• Long-form (> ~10 s) or high-resolution (> 720p) generation — the 1.3B base model is not optimized for these and quality / temporal consistency will degrade.
• Real-time or interactive generation — diffusion T2V is slow (minutes per clip).

---

🚀 Quickstart (Local Inference)

1. Install dependencies

pip install -U "diffusers>=0.31.0" transformers accelerate torch imageio imageio-ffmpeg

2. Load the base pipeline + LoRA adapter and generate a video

import torch
from diffusers import WanPipeline
from diffusers.utils import export_to_video

BASE_MODEL = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
LORA_REPO  = "Varun95/t2v-finetune-lora"

# Load the base pipeline in fp16 (T4 / 16 GB VRAM friendly)
pipe = WanPipeline.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)

# Attach the LoRA adapter from this repo (weights auto-downloaded from the Hub)
pipe.load_lora_weights(LORA_REPO)

# --- Memory optimizations (essential on a T4 / free-tier GPU) ---
pipe.enable_model_cpu_offload()  # keep weights in RAM, move to GPU only during inference
pipe.enable_vae_slicing()        # decode VAE in slices to cut VRAM during video decode
# pipe.enable_vae_tiling()        # optional: further VRAM savings at a small quality cost

prompt = "A cinematic tracking shot of a futuristic city at night, neon lights reflecting on wet rain-soaked streets, volumetric fog"

output = pipe(
    prompt=prompt,
    num_frames=49,
    height=480,
    width=832,
    num_inference_steps=30,
    guidance_scale=5.0,
    negative_prompt="blurry, low quality, distorted, watermark, text, cartoon",
    generator=torch.Generator("cuda").manual_seed(42),
).frames[0]

export_to_video(output, "output.mp4", fps=8)
print("Saved output.mp4")

💡 On a T4 (16 GB) the config above (480×832, 49 frames, 30 steps) runs in ≈ 90–150 s after model load. Reduce num_frames to 25 and num_inference_steps to 20 for a faster (~45 s) draft.

3. Merge the LoRA permanently (optional)

If you ship a single merged checkpoint (faster cold-start, no adapter overhead):

pipe = WanPipeline.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
pipe.load_lora_weights(LORA_REPO)
pipe.fuse_lora(lora_scale=1.0)          # merge weights into the base
pipe.unload_lora_weights()              # drop the now-redundant adapter
pipe.save_pretrained("./wan2.1-t2v-1.3b-merged-fp16")

4. Adjust LoRA strength

pipe.load_lora_weights(LORA_REPO)
pipe.set_adapters("default", adapter_weights=0.8)  # 0.0 = base, 1.0 = full LoRA

---

⚙️ Generation Parameter Reference

Parameter	Type	Default	Range	Notes
prompt	str	—	any	Natural-language video description.
negative_prompt	str	""	any	Concepts to avoid (blurry, watermark…).
num_frames	int	49	25–81	More frames = longer clip & more VRAM. 49 @ 8 fps ≈ 6 s.
height / width	int	480 / 832	256–1024	Keep aspect ≈ 16:9 (832×480) or 9:16.
num_inference_steps	int	30	20–50	Higher = better quality, slower.
guidance_scale	float	5.0	3.0–8.0	CFG strength; too high → artifacts.
fps (export)	int	8	4–24	Output video frame rate.
generator	torch.Generator	—	—	Set a seed for reproducibility.

---

🧪 Fine-tuning Details (Training Recipe)

This adapter was trained with diffusers + peft LoRA. Representative config:

# wan_t2v_lora_train.yaml
base_model: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
mixed_precision: bf16          # use fp16 only if bf16 is unavailable
gradient_checkpointing: true

lora:
  rank: 64
  alpha: 128
  dropout: 0.05
  target_modules:            # attention + feed-forward projections
    - "to_q"
    - "to_k"
    - "to_v"
    - "to_out.0"
    - "ff.net.0.proj"
    - "ff.net.2"

optimizer: adamw
learning_rate: 1.0e-4
lr_scheduler: cosine
warmup_steps: 500
batch_size: 1
gradient_accumulation_steps: 4
max_train_steps: 5000
save_steps: 500
resolution: "480,832"
max_num_frames: 49

Saving the trained adapter back to the Hub:

from huggingface_hub import HfApi
pipe.save_lora_weights("./lora_weights")          # saves adapter_model.safetensors
HfApi().upload_folder(
    folder_path="./lora_weights",
    repo_id="Varun95/t2v-finetune-lora",
    repo_type="model",
)

📝 Note: confirm the exact rank/alpha/target_modules and dataset against your training run before publishing — the values above reflect the default recipe used for this adapter class.

---

🌐 Serving as an API (Google Colab T4 + FastAPI + Cloudflare Tunnel)

Text-to-video generation takes 1–3 minutes per clip on a T4, so a synchronous HTTP request will time out. The recipe below uses an async job queue: the client posts a prompt, immediately receives a job_id, and polls a /status endpoint until the clip is ready.

Cell 1 — Install dependencies

!pip install -q "diffusers>=0.31.0" transformers accelerate fastapi uvicorn pillow imageio imageio-ffmpeg nest_asyncio
!wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb
!dpkg -i cloudflared-linux-amd64.deb

Cell 2 — Load the base model + LoRA adapter (T4-safe)

import torch, os
from diffusers import WanPipeline

print("Loading Wan 2.1 1.3B + LoRA adapter... (≈ 3 min cold start)")
BASE_MODEL = "Wan-AI/Wan2.1-T2V-1.3B-Diffusers"
LORA_REPO  = "Varun95/t2v-finetune-lora"

pipe = WanPipeline.from_pretrained(BASE_MODEL, torch_dtype=torch.float16)
pipe.load_lora_weights(LORA_REPO)          # apply this fine-tune
pipe.enable_model_cpu_offload()            # keeps weights in RAM, GPU-only at inference
pipe.enable_vae_slicing()                  # cut VRAM during video decode

os.makedirs("/content/outputs", exist_ok=True)
print("Model + LoRA loaded.")

Cell 3 — FastAPI server + background job queue + Cloudflare Tunnel

import nest_asyncio, threading, uuid, subprocess, re
from fastapi import FastAPI, BackgroundTasks
from fastapi.middleware.cors import CORSMiddleware
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
from diffusers.utils import export_to_video
import uvicorn

nest_asyncio.apply()
app = FastAPI(title="Wan 2.1 T2V (LoRA) API")
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"], allow_methods=["*"], allow_headers=["*"],
)
app.mount("/videos", StaticFiles(directory="/content/outputs"), name="videos")

jobs = {}   # in-memory job store: job_id -> {status, video_url, error, ...}

class VideoRequest(BaseModel):
    prompt: str
    num_frames: int = 49
    height: int = 480
    width: int = 832
    num_inference_steps: int = 30
    guidance_scale: float = 5.0
    negative_prompt: str = "blurry, low quality, distorted, watermark, text"

def generate_video_task(job_id, req: VideoRequest):
    jobs[job_id]["status"] = "processing"
    try:
        frames = pipe(
            prompt=req.prompt,
            negative_prompt=req.negative_prompt,
            num_frames=req.num_frames,
            height=req.height,
            width=req.width,
            num_inference_steps=req.num_inference_steps,
            guidance_scale=req.guidance_scale,
        ).frames[0]
        path = f"/content/outputs/{job_id}.mp4"
        export_to_video(frames, path, fps=8)
        jobs[job_id].update(status="completed", video_url=f"/videos/{job_id}.mp4")
    except Exception as e:
        jobs[job_id].update(status="failed", error=str(e))

@app.post("/generate")
async def generate(req: VideoRequest, bg: BackgroundTasks):
    job_id = str(uuid.uuid4())
    jobs[job_id] = {"status": "pending"}
    bg.add_task(generate_video_task, job_id, req)
    return {"job_id": job_id, "status": "pending"}

@app.get("/status/{job_id}")
async def status(job_id: str):
    return jobs.get(job_id, {"error": "Job not found"})

@app.get("/health")
async def health():
    return {"status": "ok", "model_loaded": pipe is not None}

# Run Uvicorn in a background thread
threading.Thread(
    target=lambda: uvicorn.run(app, host="0.0.0.0", port=8000, log_level="error"),
    daemon=True,
).start()
print("FastAPI listening on http://localhost:8000")

# Start Cloudflare Tunnel and capture the public URL
print("Starting Cloudflare Tunnel...")
proc = subprocess.Popen(
    ["cloudflared", "tunnel", "--url", "http://localhost:8000"],
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True,
)
for line in proc.stdout:
    print(line, end="")
    m = re.search(r"https://[a-zA-Z0-9-]+\.trycloudflare\.com", line)
    if m:
        print(f"\nPUBLIC API URL: {m.group(0)}\nPut this in NEXT_PUBLIC_API_URL")
        break

After the tunnel prints a URL like https://<random>.trycloudflare.com, set it as your frontend's NEXT_PUBLIC_API_URL. Keep the Colab tab open — closing it stops the tunnel.

API contract

Method	Path	Body	Returns
POST	/generate	{prompt, num_frames?, height?, width?, num_inference_steps?, guidance_scale?, negative_prompt?}	{job_id, status:"pending"}
GET	/status/{job_id}	—	{status: "pending"|"processing"|"completed"|"failed", video_url?, error?}
GET	/videos/{file}.mp4	—	the MP4 file (static)
GET	/health	—	{status:"ok", model_loaded:bool}

---

🖥️ Full-Stack Integration: Next.js Frontend (Polling)

A production frontend must not hold the HTTP connection open for minutes. Instead it posts the prompt, gets a job_id, and polls /status every few seconds.

Project setup

npx create-next-app@latest wan-t2v-studio --ts --tailwind --app
cd wan-t2v-studio
npx shadcn@latest init
npm i framer-motion lucide-react

.env.local

# The Cloudflare Tunnel URL printed by Colab Cell 3
NEXT_PUBLIC_API_URL=https://<random>.trycloudflare.com

src/services/api.ts — typed client with polling

const API_URL = process.env.NEXT_PUBLIC_API_URL!;

export interface GenerateParams {
  prompt: string;
  num_frames?: number;
  height?: number;
  width?: number;
  num_inference_steps?: number;
  guidance_scale?: number;
  negative_prompt?: string;
}

export interface JobStatus {
  status: "pending" | "processing" | "completed" | "failed";
  video_url?: string;
  error?: string;
}

export async function requestGeneration(params: GenerateParams): Promise<string> {
  const res = await fetch(`${API_URL}/generate`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify(params),
  });
  if (!res.ok) throw new Error(`Generate failed: ${res.status}`);
  const data = await res.json();
  return data.job_id as string;
}

export async function pollStatus(
  jobId: string,
  opts: { intervalMs?: number; timeoutMs?: number; onUpdate?: (s: JobStatus) => void } = {}
): Promise<JobStatus> {
  const { intervalMs = 5000, timeoutMs = 5 * 60 * 1000, onUpdate } = opts;
  const start = Date.now();

  while (Date.now() - start < timeoutMs) {
    const res = await fetch(`${API_URL}/status/${jobId}`);
    const status: JobStatus = await res.json();
    onUpdate?.(status);
    if (status.status === "completed" || status.status === "failed") return status;
    await new Promise((r) => setTimeout(r, intervalMs));
  }
  throw new Error("Polling timed out");
}

export function videoUrl(path: string): string {
  return path.startsWith("http") ? path : `${API_URL}${path}`;
}

src/app/page.tsx — cinematic UI (Tailwind + Framer Motion)

"use client";
import { useState } from "react";
import { motion, AnimatePresence } from "framer-motion";
import { Loader2, Sparkles, Film } from "lucide-react";
import { requestGeneration, pollStatus, videoUrl, type JobStatus } from "@/services/api";

export default function Home() {
  const [prompt, setPrompt] = useState("A cinematic shot of a futuristic city at night, neon lights");
  const [numFrames, setNumFrames] = useState(49);
  const [guidance, setGuidance] = useState(5.0);
  const [status, setStatus] = useState<JobStatus | null>(null);
  const [busy, setBusy] = useState(false);
  const [history, setHistory] = useState<string[]>([]);

  async function handleGenerate() {
    if (!prompt.trim() || busy) return;
    setBusy(true); setStatus({ status: "pending" });
    try {
      const jobId = await requestGeneration({
        prompt, num_frames: numFrames, guidance_scale: guidance,
      });
      const final = await pollStatus(jobId, { onUpdate: setStatus });
      setStatus(final);
      if (final.status === "completed" && final.video_url) {
        setHistory((h) => [videoUrl(final.video_url!), ...h].slice(0, 12));
      }
    } catch (e: unknown) {
      setStatus({ status: "failed", error: (e as Error).message });
    } finally { setBusy(false); }
  }

  const statusLabel = {
    pending: "Queued", processing: "Rendering on GPU",
    completed: "Done", failed: "Failed",
  } as const;

  return (
    <main className="min-h-screen bg-[#0a0e1a] text-slate-100 p-8">
      <header className="max-w-5xl mx-auto mb-10">
        <h1 className="text-4xl font-bold bg-gradient-to-r from-teal-400 to-violet-400 bg-clip-text text-transparent">
          Wan T2V Studio
        </h1>
        <p className="text-slate-400 mt-2">Text-to-video generation · Wan 2.1 1.3B + LoRA</p>
      </header>

      <section className="max-w-5xl mx-auto grid md:grid-cols-3 gap-6">
        <div className="md:col-span-2 space-y-4 p-6 rounded-2xl bg-white/5 backdrop-blur border border-white/10">
          <textarea
            value={prompt} onChange={(e) => setPrompt(e.target.value)}
            rows={4}
            className="w-full p-4 rounded-xl bg-black/40 border border-white/10 focus:ring-2 ring-teal-400 outline-none"
            placeholder="Describe your video"
          />
          <div className="flex gap-6 text-sm">
            <label className="flex-1">Frames
              <input type="range" min={25} max={81} value={numFrames}
                onChange={(e) => setNumFrames(+e.target.value)} className="w-full accent-teal-400" />
              <span className="text-slate-400">{numFrames}</span>
            </label>
            <label className="flex-1">Guidance
              <input type="range" min={3} max={8} step={0.5} value={guidance}
                onChange={(e) => setGuidance(+e.target.value)} className="w-full accent-violet-400" />
              <span className="text-slate-400">{guidance}</span>
            </label>
          </div>
          <button onClick={handleGenerate} disabled={busy}
            className="w-full py-3 rounded-xl bg-gradient-to-r from-teal-500 to-violet-500 font-semibold disabled:opacity-50 flex items-center justify-center gap-2">
            {busy ? <Loader2 className="animate-spin" /> : <Sparkles />} Generate Video
          </button>
        </div>

        <div className="p-6 rounded-2xl bg-white/5 backdrop-blur border border-white/10">
          <AnimatePresence mode="wait">
            <motion.div key={status?.status}
              initial={{ opacity: 0, y: 8 }} animate={{ opacity: 1, y: 0 }}>
              <p className="text-slate-400 text-sm">Status</p>
              <p className="text-xl font-semibold">
                {status ? statusLabel[status.status] : "Idle"}
              </p>
              {status?.status === "completed" && status.video_url && (
                <video src={videoUrl(status.video_url)} controls autoPlay loop
                  className="mt-4 w-full rounded-xl" />
              )}
              {status?.status === "failed" && (
                <p className="text-red-400 text-sm mt-2">{status.error}</p>
              )}
            </motion.div>
          </AnimatePresence>
        </div>
      </section>

      {history.length > 0 && (
        <section className="max-w-5xl mx-auto mt-10">
          <h2 className="text-lg font-semibold mb-4 flex items-center gap-2"><Film /> History</h2>
          <div className="grid grid-cols-2 md:grid-cols-4 gap-4">
            {history.map((src, i) => (
              <video key={i} src={src} controls
                className="w-full rounded-xl border border-white/10" />
            ))}
          </div>
        </section>
      )}
    </main>
  );
}

Run the frontend

npm run dev   # http://localhost:3000

---

🧭 End-to-End Deployment Checklist

1. Colab backend — run Cells 1–3 (Runtime → GPU T4). Copy the printed https://<random>.trycloudflare.com URL.
2. Frontend — copy .env.local and set NEXT_PUBLIC_API_URL to the tunnel URL, then npm run dev.
3. Verify — open the app, submit a prompt, watch the status flip pending → processing → completed, then play the returned MP4.
4. Keep Colab alive — the tunnel dies when the Colab tab / runtime closes.

---

⚠️ Limitations & Bias

• Resolution / length caps — best results at 480×832, ≤ 49 frames; longer clips degrade in temporal coherence.
• VRAM bound — needs CPU offloading on ≤ 16 GB GPUs; a 24 GB+ GPU enables larger resolutions and more frames.
• Prompt sensitivity — complex multi-subject prompts can produce fused or warped subjects; use a strong negative_prompt.
• No audio — output is silent video; pair with a separate audio model if needed.
• Inherited bias — the adapter inherits the base model's biases (demographic, cultural, stylistic). Audit outputs before public deployment.
• Hallucinated realism — can produce convincing but fabricated scenes; do not use for evidence, journalism, or documentation without disclosure.

---

🌱 Environmental Impact

• Training — LoRA fine-tuning of a 1.3B diffusion model for ~5k steps on a single A100 is roughly 20–40 GPU-hours (≈ 5–10 kg CO₂eq depending on grid mix).
• Inference — ≈ 90–150 s on a T4 per 49-frame clip; roughly 0.005–0.01 kWh per generation. Batch and cache where possible.

Estimates are approximate; measure on your own hardware for accurate accounting.

---

📚 Citation

If you use this adapter, please cite both this repo and the original Wan 2.1 work:

@misc{varun95_t2v_lora,
  author       = {Varun95},
  title        = {t2v-finetune-lora: LoRA fine-tune of Wan2.1-T2V-1.3B-Diffusers},
  year         = {2025},
  url          = {https://huggingface.co/Varun95/t2v-finetune-lora}
}

@misc{wan21,
  title        = {Wan2.1: Open and Advanced Large-Scale Video Generative Models},
  author       = {Wan-AI},
  year         = {2025},
  url          = {https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers}
}

---

🙏 Acknowledgements

• Wan-AI for the open Wan 2.1 base model.
• Diffusers for the WanPipeline and LoRA tooling.
• Cloudflare for the free trycloudflare tunnel.

---

License

MIT. The underlying Wan 2.1 base model has its own license; respect it when redistributing merged checkpoints.