Tutorial on training LoRA for Hunyuan Video on Windows, but I need it to run on Modal due to the juicy L4 GPU! Help me!
```
# Training a LoRA for Hunyuan Video on Windows

**Version 0.2**

**Updates:**

* Adjusted epochs/image repeats. **Please adjust these depending on your data size.** For my style LoRA, I used 2500-3000 steps total.
* Added info about checkpoint resuming.

**Version 0.1**

**Feedback Welcome:** I already had a WSL environment set up, so some steps may be incorrect!

**Disclaimer:** This process is fairly complex. If this seems too difficult, consider waiting for a native Windows tool. I haven't experimented extensively with training settings, but these worked for me for a style LoRA.

**Target Audience:** This guide is aimed at using images to train the LoRA. Small modifications will be needed to train with video.

---

## Initial Setup

### 1. WSL Setup

* **Open Windows PowerShell as Administrator:**
```powershell
wsl --install
```

* **After installation:**
1. Restart your computer.
2. Ubuntu will automatically start and prompt you to:
* Create a username.
* Set a password.

**Important:** Remember these credentials; you'll need them for `sudo` commands.

### 2. Ubuntu Initial Setup

* **Update package lists:**

```bash
sudo apt update
sudo apt full-upgrade -y
```

* **Install dependencies:**

```bash
sudo apt-get install -y git-lfs wget python3-dev build-essential
```

### 3. Verify NVIDIA Setup

* **Check NVIDIA drivers:**

```bash
nvidia-smi
```

This should display your GPU(s) and driver version.

* **Troubleshooting:** If this command fails, you may need to install the NVIDIA CUDA driver for WSL:
* Download from: [https://developer.nvidia.com/cuda/wsl](https://developer.nvidia.com/cuda/wsl)

### 4. Miniconda Installation

```bash
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b
source ~/.bashrc
```

### 5. Clone Training Repository

```bash
git clone --recurse-submodules https://github.com/tdrussell/diffusion-pipe
cd diffusion-pipe
```

### 6. Setup Python Environment

```bash
conda create -n diffusion-pipe python=3.12
conda activate diffusion-pipe
```

**Important:** Install PyTorch *before* installing `requirements.txt`. This is a potential area for issues.

```bash
pip install torch==2.4.1 torchvision==0.19.1 --index-url https://download.pytorch.org/whl/cu121
pip install torchaudio==2.4.1+cu121 --index-url https://download.pytorch.org/whl/cu121
```
* **Note:** These versions worked for me, but your mileage may vary (YMMV).

* **Install requirements:**

```bash
pip install -r requirements.txt
```

* **Troubleshooting:**
* **CUDA Compilation Errors:** If you encounter errors during the `pip install` of DeepSpeed or other packages, you might need to install `nvidia-cuda-toolkit` via `apt`.
* **Other `pip`/`torch` Errors:** Consult your favorite LLM or online resources for solutions.

---

## Accessing Files in Windows

Easily access your WSL files in Windows File Explorer:

1. Navigate to:

```
\\wsl$\Ubuntu\home\\diffusion-pipe\
```

Replace `` with your WSL username.

This allows easy transfer of images to your training folder and copying the finished LoRA to ComfyUI.

---

## Download and Organize Models

**If you have existing files, copy them from Windows to the specified folders.**

**Otherwise:**

```bash
cd ~/diffusion-pipe
mkdir -p models/{hunyuan,clip,llm}
```

* **Download HunyuanVideo files:**

```bash
wget https://huggingface.co/Kijai/HunyuanVideo_comfy/resolve/main/hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors -P ~/diffusion-pipe/models/hunyuan/
wget https://huggingface.co/Kijai/HunyuanVideo_comfy/resolve/main/hunyuan_video_vae_bf16.safetensors -P ~/diffusion-pipe/models/hunyuan/
```

* **Download CLIP model:**

```bash
git clone https://huggingface.co/openai/clip-vit-large-patch14 models/clip
```

* **Download LLM:**

```bash
git clone https://huggingface.co/Kijai/llava-llama-3-8b-text-encoder-tokenizer models/llm
```

---

## Configuration Files

Create two configuration files in the main directory (`diffusion-pipe`): `config.toml` and `dataset.toml`.

### 1. Training Configuration (`config.toml`)

**Example `config.toml` (adjust as needed):**

```toml
# Dataset config file.
output_dir = '~/training_output'
dataset = 'dataset.toml'

# Training settings
epochs = 50
micro_batch_size_per_gpu = 1
pipeline_stages = 1
gradient_accumulation_steps = 4
gradient_clipping = 1.0
warmup_steps = 100

# eval settings
eval_every_n_epochs = 5
eval_before_first_step = true
eval_micro_batch_size_per_gpu = 1
eval_gradient_accumulation_steps = 1

# misc settings
save_every_n_epochs = 5
checkpoint_every_n_minutes = 30
activation_checkpointing = true
partition_method = 'parameters'
save_dtype = 'bfloat16'
caching_batch_size = 1
steps_per_print = 1
video_clip_mode = 'single_middle'

[model]
type = 'hunyuan-video'
transformer_path = '~/diffusion-pipe/models/hunyuan/hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors'
vae_path = '~/diffusion-pipe/models/hunyuan/hunyuan_video_vae_bf16.safetensors'
llm_path = '~/diffusion-pipe/models/llm'
clip_path = '~/diffusion-pipe/models/clip'
dtype = 'bfloat16'
transformer_dtype = 'float8'
timestep_sample_method = 'logit_normal'

[adapter]
type = 'lora'
rank = 64
dtype = 'bfloat16'

[optimizer]
type = 'adamw_optimi'
lr = 5e-5
betas = [0.9, 0.99]
weight_decay = 0.02
eps = 1e-8
```

### 2. Dataset Configuration (`dataset.toml`)

**Example `dataset.toml`:**

```toml
# Resolution settings.
# Can adjust this to 1024 for image training, especially on 24gb cards.
resolutions = [512]

# Aspect ratio bucketing settings
enable_ar_bucket = true
min_ar = 0.5
max_ar = 2.0
num_ar_buckets = 7

# Frame buckets (1 is for images)
frame_buckets = [1]

[[directory]]
# Set this to where your dataset is
path = '~/training_data/images'
# Reduce as necessary
num_repeats = 5
```

---

## Preparing Training Data

1. **Create dataset directory:**

```bash
mkdir -p ~/training_data/images
```

2. **Place training images in the directory:**
* **LoRA:** 20-50 diverse images.
* **Optional:** Create matching `.txt` files with prompts (same name as the image file).

**Example structure:**

```
~/training_data/images
├── image1.png
├── image1.txt # Optional prompt file
├── image2.png
└── image2.txt
```

---

## Training

**Launch training:**

```bash
NCCL_P2P_DISABLE="1" NCCL_IB_DISABLE="1" deepspeed --num_gpus=1 train.py --deepspeed --config config.toml
```

---

## Monitoring Training

* **Monitor GPU usage (Windows terminal):**

```bash
nvidia-smi --query-gpu=timestamp,name,temperature.gpu,utilization.gpu,memory.used,memory.total --format=csv -l 5
```

* Training outputs will be saved in the directory specified by `output_dir` in your `config.toml`.

---

## Resuming from Checkpoint

**If your computer crashes or you need to shut it down:**

Use the `--resume_from_checkpoint` flag. If your GPU is slow, consider checkpointing more frequently (but be mindful of storage space).

**Example:**

```bash
NCCL_P2P_DISABLE="1" NCCL_IB_DISABLE="1" deepspeed --num_gpus=1 train.py --deepspeed --config config.toml --resume_from_checkpoint
```

---

## Using the Trained LoRA

### After training:

1. **Find your LoRA file:**
* Navigate to the training output directory in Windows:

```
\\wsl$\Ubuntu\home\\training_output
```

* Look for the latest epoch folder.
* Find the `adapter.safetensors` file.

### Using with ComfyUI:

1. Copy and rename `adapter.safetensors` (give it a descriptive name) to your ComfyUI LoRAs folder.
2. Ensure you have the **HunyuanVideoWrapper** node installed: [https://github.com/kijai/ComfyUI-HunyuanVideoWrapper](https://github.com/kijai/ComfyUI-HunyuanVideoWrapper)
3. Use the "HunyuanVideo Lora Select" node to load it.
4. Experiment with different epochs to find the optimal setting for your dataset.
```

**Modal Data Structure**

```
/
├── root/
│   └── volume/  # Modal Volume
│       ├── models/
│       │   ├── llm/
│       │   │   └── ... (model files)
│       │   ├── clip/
│       │   │   └── ... (model files)
│       │   └── hunyuan/
│       │       ├── hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors
│       │       └── hunyuan_video_vae_bf16.safetensors
│       ├── training_data/
│       │   └── images/
│       │       ├── image1.png
│       │       ├── image1.txt (Optional)
│       │       ├── ...
│       └── training_output/
│           └── 2024-07-25_14-30-00/ (Example timestamped output directory)
│               └── ... (populated during training)
├── diffusion-pipe/  # Cloned training repository
│   ├── train.py
│   ├── ... (other files from the repo)
│   ├── config.toml
│   └── dataset.toml
└── app.py  # Your Modal script
```

I have a mockup: (try not to change anything that already exists unless you're pretty sure they work)
```python
import io
import modal
import os
import subprocess
import sys
import time
import shutil
from datetime import datetime
from typing import Dict, List, Tuple

# --- Modal Setup ---
app = modal.App("hunyuan-training")
volume = modal.Volume.from_name(
    "hunyuan-volume", create_if_missing=True
)  # Persistent volume
image = modal.Image.debian_slim().apt_install(
    "git", "wget", "curl"
).pip_install(
    "tomli",  # Install tomli for parsing TOML
    "toml",  # Install toml for writing TOML
    "diffusers",
    "transformers",
    "accelerate",
    "safetensors",
    "torch>=2.0",
    "torchvision",
    "tensorboard",
    "ftfy",
    "protobuf==3.20.1",
    "deepspeed",
    "huggingface_hub[hf_transfer]",
).env({"HF_HUB_ENABLE_HF_TRANSFER": "1"})

# --- Constants ---
GIT_REPO = "https://github.com/tdrussell/diffusion-pipe.git"
TRAINING_DATA_LOCAL = "./training_data"
CONFIG_LOCAL = "./config.toml"
DATASET_LOCAL = "./dataset.toml"
REMOTE_ROOT = "/root/volume"

# --- Helper Functions ---
@app.function(image=image, volumes={REMOTE_ROOT: volume})
def hf_download_repo(repo_id: str, model_type: str, ignore_patterns: List[str] = None):
    from huggingface_hub import snapshot_download

    snapshot_download(
        repo_id=repo_id,
        local_dir=f"/root/volume/models/{model_type}",
        local_dir_use_symlinks=False,  # Crucial for Modal Volumes
        ignore_patterns=ignore_patterns or [], # Handle ignore patterns
    )
    print(f"Repository {repo_id} downloaded to /root/models/{model_type}")

@app.function(image=image, volumes={REMOTE_ROOT: volume})
def hf_download_file(repo_id: str, filename: str, model_type: str):
    from huggingface_hub import hf_hub_download

    hf_hub_download(
        repo_id=repo_id,
        filename=filename,
        local_dir=f"/root/volume/models/{model_type}",
    )
    print(f"File {filename} from {repo_id} downloaded to /root/models/{model_type}")

# --- Modal Functions ---
@app.function(image=image, volumes={REMOTE_ROOT: volume})
def download_models():
    models_to_download_from_hf_repos: List[Tuple[str, str, List[str]]] = [
        # Example: Download an entire repo
        # ("stabilityai/stable-diffusion-xl-base-1.0", "checkpoints"),
        # Add your repo_id and model_type here
        ("Kijai/llava-llama-3-8b-text-encoder-tokenizer", "llm"),
        ("openai/clip-vit-large-patch14", "clip", ["flax_model.msgpack", "pytorch_model.bin", "tf_model.h5"]), # Exclude these file patterns
    ]

    models_to_download_from_hf_files: List[Tuple[str, str, str]] = [
        # format is (huggingface repo_id, the model filename, comfyui models subdirectory)
        ("Kijai/HunyuanVideo_comfy", "hunyuan_video_720_cfgdistill_fp8_e4m3fn.safetensors", "hunyuan"),
        ("Kijai/HunyuanVideo_comfy", "hunyuan_video_vae_bf16.safetensors", "hunyuan"),
    ]

    # Download entire HF repos in parallel
    print("Downloading entire Hugging Face repositories...")
    list(hf_download_repo.starmap(models_to_download_from_hf_repos))

    # Download models from Hugging Face in parallel
    print("Downloading individual files from Hugging Face...")
    list(hf_download_file.starmap(models_to_download_from_hf_files))

@app.function(
    image=image,
    gpu="T4",
    volumes={REMOTE_ROOT: volume},
    timeout=86400,  # 24 hours
)
def train():
#TODO

# --- Entry Points ---
@app.local_entrypoint()
def train_entrypoint():
    train.remote()
```