README.md

---
base_model: runwayml/stable-diffusion-v1-5
library_name: diffusers
license: creativeml-openrail-m
tags:
- stable-diffusion
- stable-diffusion-diffusers
- text-to-image
- diffusers
- controlnet
- control-lora-v3
- diffusers-training
inference: true
---

<!-- This model card has been generated automatically according to the information the training script had access to. You
should probably proofread and complete it, then remove this comment. -->


# ControlLoRA Version 3 Pretrained Models Collection

This is a collections of control-lora-v3 weights trained on runwayml/stable-diffusion-v1-5 and stabilityai/stable-diffusion-xl-base-1.0 with different types of conditioning.
You can find some example images below.

## Stable Diffusion

### Canny
<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/canny1.png" style="height:256px;" />
  <img src="./imgs/canny2.png" style="height:256px;" />
  <img src="./imgs/canny3.png" style="height:256px;" />
  <img src="./imgs/canny4.png" style="height:256px;" />
  <img src="./imgs/canny_vermeer.png" style="height:256px;" />
</div>

### OpenPose + Segmentation

This is experimental, and it doesn't work well.

<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/pose5_segmentation5.png" style="height:256px;" />
  <img src="./imgs/pose6_segmentation6.png" style="height:256px;" />
  <img src="./imgs/pose7_segmentation7.png" style="height:256px;" />
  <img src="./imgs/pose8_segmentation8.png" style="height:256px;" />
</div>

### Depth
<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/depth1.png" style="height:256px;" />
  <img src="./imgs/depth2.png" style="height:256px;" />
  <img src="./imgs/depth3.png" style="height:256px;" />
  <img src="./imgs/depth4.png" style="height:256px;" />
</div>

### Normal map
<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/normal1.png" style="height:256px;" />
  <img src="./imgs/normal2.png" style="height:256px;" />
  <img src="./imgs/normal3.png" style="height:256px;" />
  <img src="./imgs/normal4.png" style="height:256px;" />
</div>

### OpenPose
<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/pose1.png" style="height:256px;" />
  <img src="./imgs/pose2.png" style="height:256px;" />
  <img src="./imgs/pose3.png" style="height:256px;" />
  <img src="./imgs/pose4.png" style="height:256px;" />
  <img src="./imgs/pose5.png" style="height:256px;" />
  <img src="./imgs/pose6.png" style="height:256px;" />
  <img src="./imgs/pose7.png" style="height:256px;" />
  <img src="./imgs/pose8.png" style="height:256px;" />
</div>

### Segmentation

<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/segmentation1.png" style="height:256px;" />
  <img src="./imgs/segmentation2.png" style="height:256px;" />
  <img src="./imgs/segmentation3.png" style="height:256px;" />
  <img src="./imgs/segmentation4.png" style="height:256px;" />
  <img src="./imgs/segmentation5.png" style="height:256px;" />
  <img src="./imgs/segmentation6.png" style="height:256px;" />
  <img src="./imgs/segmentation7.png" style="height:256px;" />
  <img src="./imgs/segmentation8.png" style="height:256px;" />
</div>

### Tile
<div style="display: flex; flex-wrap: wrap;">
  <img src="./imgs/tile1.png" style="height:256px;" />
  <img src="./imgs/tile2.png" style="height:256px;" />
  <img src="./imgs/tile3.png" style="height:256px;" />
  <img src="./imgs/tile4.png" style="height:256px;" />
</div>

## Stable Diffusion XL

### Canny
<div style="display: flex;">
  <img src="./imgs/sdxl_canny1.png" style="height:256px;" />
  <img src="./imgs/sdxl_canny2.png" style="height:256px;" />
  <img src="./imgs/sdxl_canny3.png" style="height:256px;" />
  <img src="./imgs/sdxl_canny4.png" style="height:256px;" />
  <img src="./imgs/sdxl_canny_vermeer.png" style="height:256px;" />
</div>

## Intended uses & limitations

#### How to use

First clone the [control-lora-v3](https://github.com/HighCWu/control-lora-v3) and `cd` in the directory:
```sh
git clone https://github.com/HighCWu/control-lora-v3
cd control-lora-v3
```

Then run the python code。

For stable diffusion, use:

```py
# !pip install opencv-python transformers accelerate
from diffusers import UniPCMultistepScheduler
from diffusers.utils import load_image
from model import UNet2DConditionModelEx
from pipeline import StableDiffusionControlLoraV3Pipeline
import numpy as np
import torch

import cv2
from PIL import Image

# download an image
image = load_image(
    "https://hf.co/datasets/huggingface/documentation-images/resolve/main/diffusers/input_image_vermeer.png"
)
image = np.array(image)

# get canny image
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# load stable diffusion v1-5 and control-lora-v3 
unet: UNet2DConditionModelEx = UNet2DConditionModelEx.from_pretrained(
    "runwayml/stable-diffusion-v1-5", subfolder="unet", torch_dtype=torch.float16
)
unet = unet.add_extra_conditions(["canny"])
pipe = StableDiffusionControlLoraV3Pipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", unet=unet, torch_dtype=torch.float16
)
# load attention processors
# pipe.load_lora_weights("HighCWu/sd-control-lora-v3-canny")
pipe.load_lora_weights("HighCWu/control-lora-v3", subfolder="sd-control-lora-v3-canny-half_skip_attn-rank16-conv_in-rank64")

# speed up diffusion process with faster scheduler and memory optimization
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
# remove following line if xformers is not installed
pipe.enable_xformers_memory_efficient_attention()

pipe.enable_model_cpu_offload()

# generate image
generator = torch.manual_seed(0)
image = pipe(
    "futuristic-looking woman", num_inference_steps=20, generator=generator, image=canny_image
).images[0]
image.show()
```

For stable diffusion xl, use:

```py
# !pip install opencv-python transformers accelerate
from diffusers import AutoencoderKL
from diffusers.utils import load_image
from model import UNet2DConditionModelEx
from pipeline_sdxl import StableDiffusionXLControlLoraV3Pipeline
import numpy as np
import torch

import cv2
from PIL import Image

prompt = "aerial view, a futuristic research complex in a bright foggy jungle, hard lighting"
negative_prompt = "low quality, bad quality, sketches"

# download an image
image = load_image(
    "https://hf.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png"
)

# initialize the models and pipeline
unet: UNet2DConditionModelEx = UNet2DConditionModelEx.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet", torch_dtype=torch.float16
)
unet = unet.add_extra_conditions(["canny"])
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlLoraV3Pipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", unet=unet, vae=vae, torch_dtype=torch.float16
)
# load attention processors
# pipe.load_lora_weights("HighCWu/sdxl-control-lora-v3-canny")
pipe.load_lora_weights("HighCWu/control-lora-v3", subfolder="sdxl-control-lora-v3-canny-half_skip_attn-rank16-conv_in-rank64")
pipe.enable_model_cpu_offload()

# get canny image
image = np.array(image)
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
canny_image = Image.fromarray(image)

# generate image
image = pipe(
    prompt, image=canny_image
).images[0]
image.show()
```

#### Limitations and bias

[TODO: provide examples of latent issues and potential remediations]

## Training details

[TODO: describe the data used to train the model]