LoRA text2image fine-tuning - Miracle-2001/japanese-anime-lora

These are LoRA adaption weights for runwayml/stable-diffusion-v1-5. The weights were fine-tuned on the mintz1104/diffusion_stage_design_japanese_anime_style dataset. You can find some example images in the following.

Intended uses & limitations

How to use

export LORA_PATH="Miracle-2001/diagram-lora"
export OUTPUT_NAME="MLP.png"

HF_ENDPOINT=https://hf-mirror.com python inference.py \
  --lora_path $LORA_PATH  --output_name $OUTPUT_NAME \
  --prompt="Draw a picture of Multilayer Perceptron"

import torch
from diffusers import StableDiffusionPipeline, DPMSolverMultistepScheduler
from huggingface_hub import model_info
import argparse

parser = argparse.ArgumentParser()
parser.add_argument('--lora_path', type=str, 
                    help='lora参数在huggingface上的位置')
parser.add_argument('--prompt', type=str, 
                    help='提示词')
parser.add_argument('--output_name', type=str, 
                    help='图片的输出名称，包含扩展名。建议以.png结尾')

args = parser.parse_args()
# LoRA weights ~3 MB
model_path = args.lora_path

info = model_info(model_path)
model_base = info.cardData["base_model"]
pipe = StableDiffusionPipeline.from_pretrained(model_base, torch_dtype=torch.float16)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

pipe.unet.load_attn_procs(model_path)
pipe.to("cuda")

image = pipe(args.prompt, num_inference_steps=25).images[0]
image.save(args.output_name)

Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

Training details

# git clone https://github.com/huggingface/diffusers
# cd diffusers
# pip install .
cd ..
cd diffusers/examples/text_to_image
# pip install -r requirements.txt
# accelerate config

# huggingface-cli login

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="/flash2/aml/kyzhang24/HW3/hw3-base/japanese_cartoon_style"
export HUB_MODEL_ID="japanese-anime-lora"
# export DATASET_NAME="lambdalabs/pokemon-blip-captions"
export DATASET_NAME="mintz1104/diffusion_stage_design_japanese_anime_style"

HF_ENDPOINT=https://hf-mirror.com python train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME \
  --dataloader_num_workers=8 \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=200 \
  --learning_rate=1e-04 \
  --max_grad_norm=1 \
  --lr_scheduler="cosine" --lr_warmup_steps=0 \
  --output_dir=${OUTPUT_DIR} \
  --push_to_hub \
  --hub_model_id=${HUB_MODEL_ID} \
  --report_to=wandb \
  --checkpointing_steps=2500 \
  --validation_prompt="This is a futuristic-themed performance venue. In the center, there is a large hexagonal stage surrounded by dazzling neon lights. The backdrop features a massive electronic display." \
  --seed=1337 \
  --caption_column="text"