update model card
#1
by
williamberman
- opened
- README.md +99 -1
- images/canny.png +0 -0
- images/canny_input.png +0 -0
- images/canny_out.png +0 -0
README.md
CHANGED
@@ -1,3 +1,101 @@
|
|
1 |
---
|
2 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
+
license: apache-2.0
|
3 |
+
base_model: runwayml/stable-diffusion-v1-5
|
4 |
+
tags:
|
5 |
+
- art
|
6 |
+
- t2i-adapter
|
7 |
+
- controlnet
|
8 |
+
- stable-diffusion
|
9 |
+
- image-to-image
|
10 |
---
|
11 |
+
|
12 |
+
# T2I Adapter - Canny
|
13 |
+
|
14 |
+
T2I Adapter is a network providing additional conditioning to stable diffusion. Each t2i checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint.
|
15 |
+
|
16 |
+
This checkpoint provides conditioning on canny edge's for the stable diffusion 1.4 checkpoint.
|
17 |
+
|
18 |
+
## Model Details
|
19 |
+
- **Developed by:** T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
|
20 |
+
- **Model type:** Diffusion-based text-to-image generation model
|
21 |
+
- **Language(s):** English
|
22 |
+
- **License:** Apache 2.0
|
23 |
+
- **Resources for more information:** [GitHub Repository](https://github.com/TencentARC/T2I-Adapter), [Paper](https://arxiv.org/abs/2302.08453).
|
24 |
+
- **Cite as:**
|
25 |
+
|
26 |
+
@misc{
|
27 |
+
title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models},
|
28 |
+
author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie},
|
29 |
+
year={2023},
|
30 |
+
eprint={2302.08453},
|
31 |
+
archivePrefix={arXiv},
|
32 |
+
primaryClass={cs.CV}
|
33 |
+
}
|
34 |
+
|
35 |
+
### Checkpoints
|
36 |
+
|
37 |
+
| Model Name | Control Image Overview| Control Image Example | Generated Image Example |
|
38 |
+
|---|---|---|---|
|
39 |
+
|[TencentARC/t2iadapter_color_sd14v1](https://huggingface.co/TencentARC/t2iadapter_color_sd14v1)<br/> *Trained with spatial color palette* | A image with 8x8 color palette.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/color_sample_input.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/color_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/color_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/color_sample_output.png"/></a>|
|
40 |
+
|[TencentARC/t2iadapter_canny_sd14v1](https://huggingface.co/TencentARC/t2iadapter_canny_sd14v1)<br/> *Trained with canny edge detection* | A monochrome image with white edges on a black background.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/canny_sample_input.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/canny_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/canny_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/canny_sample_output.png"/></a>|
|
41 |
+
|[TencentARC/t2iadapter_sketch_sd14v1](https://huggingface.co/TencentARC/t2iadapter_sketch_sd14v1)<br/> *Trained with [PidiNet](https://github.com/zhuoinoulu/pidinet) edge detection* | A hand-drawn monochrome image with white outlines on a black background.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/sketch_sample_input.png"><img width="64" style="margin:0;padding:0;" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/sketch_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/sketch_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/sketch_sample_output.png"/></a>|
|
42 |
+
|[TencentARC/t2iadapter_depth_sd14v1](https://huggingface.co/TencentARC/t2iadapter_depth_sd14v1)<br/> *Trained with Midas depth estimation* | A grayscale image with black representing deep areas and white representing shallow areas.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/depth_sample_input.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/depth_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/depth_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/depth_sample_output.png"/></a>|
|
43 |
+
|[TencentARC/t2iadapter_openpose_sd14v1](https://huggingface.co/TencentARC/t2iadapter_openpose_sd14v1)<br/> *Trained with OpenPose bone image* | A [OpenPose bone](https://github.com/CMU-Perceptual-Computing-Lab/openpose) image.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/openpose_sample_input.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/openpose_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/openpose_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/openpose_sample_output.png"/></a>|
|
44 |
+
|[TencentARC/t2iadapter_keypose_sd14v1](https://huggingface.co/TencentARC/t2iadapter_keypose_sd14v1)<br/> *Trained with mmpose skeleton image* | A [mmpose skeleton](https://github.com/open-mmlab/mmpose) image.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/keypose_sample_input.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/keypose_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/keypose_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/keypose_sample_output.png"/></a>|
|
45 |
+
|[TencentARC/t2iadapter_seg_sd14v1](https://huggingface.co/TencentARC/t2iadapter_seg_sd14v1)<br/>*Trained with semantic segmentation* | An [custom](https://github.com/TencentARC/T2I-Adapter/discussions/25) segmentation protocol image.|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/seg_sample_input.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/seg_sample_input.png"/></a>|<a href="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/seg_sample_output.png"><img width="64" src="https://huggingface.co/datasets/diffusers/docs-images/resolve/main/t2i-adapter/seg_sample_output.png"/></a> |
|
46 |
+
|[TencentARC/t2iadapter_canny_sd15v2](https://huggingface.co/TencentARC/t2iadapter_canny_sd15v2)||
|
47 |
+
|[TencentARC/t2iadapter_depth_sd15v2](https://huggingface.co/TencentARC/t2iadapter_depth_sd15v2)||
|
48 |
+
|[TencentARC/t2iadapter_sketch_sd15v2](https://huggingface.co/TencentARC/t2iadapter_sketch_sd15v2)||
|
49 |
+
|[TencentARC/t2iadapter_zoedepth_sd15v1](https://huggingface.co/TencentARC/t2iadapter_zoedepth_sd15v1)||
|
50 |
+
|
51 |
+
## Example
|
52 |
+
|
53 |
+
1. Dependencies
|
54 |
+
|
55 |
+
```sh
|
56 |
+
pip install diffusers transformers opencv-contrib-python
|
57 |
+
```
|
58 |
+
|
59 |
+
2. Run code:
|
60 |
+
|
61 |
+
```python
|
62 |
+
import cv2
|
63 |
+
from PIL import Image
|
64 |
+
import torch
|
65 |
+
import numpy as np
|
66 |
+
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline
|
67 |
+
|
68 |
+
image = Image.open('./images/canny_input.png')
|
69 |
+
image = np.array(image)
|
70 |
+
|
71 |
+
low_threshold = 100
|
72 |
+
high_threshold = 200
|
73 |
+
|
74 |
+
image = cv2.Canny(image, low_threshold, high_threshold)
|
75 |
+
image = Image.fromarray(image)
|
76 |
+
|
77 |
+
image.save('./images/canny.png')
|
78 |
+
|
79 |
+
adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_canny_sd14v1", torch_dtype=torch.float16)
|
80 |
+
pipe = StableDiffusionAdapterPipeline.from_pretrained(
|
81 |
+
"CompVis/stable-diffusion-v1-4",
|
82 |
+
adapter=adapter,
|
83 |
+
torch_dtype=torch.float16,
|
84 |
+
)
|
85 |
+
pipe.to("cuda")
|
86 |
+
|
87 |
+
generator = torch.manual_seed(0)
|
88 |
+
|
89 |
+
out_image = pipe(
|
90 |
+
"a rabbit wearing glasses",
|
91 |
+
image=image,
|
92 |
+
generator=generator,
|
93 |
+
).images[0]
|
94 |
+
|
95 |
+
out_image.save('./images/canny_out.png')
|
96 |
+
```
|
97 |
+
|
98 |
+
|
99 |
+
![canny_input](./images/canny_input.png)
|
100 |
+
![canny](./images/canny.png)
|
101 |
+
![canny_out](./images/canny_out.png)
|
images/canny.png
ADDED
images/canny_input.png
ADDED
images/canny_out.png
ADDED