roygan commited on
Commit
ea78d66
1 Parent(s): 6705718

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +14 -17
README.md CHANGED
@@ -42,9 +42,21 @@ The first open source Chinese&English Bilingual Stable diffusion, which was trai
42
 
43
  ## 模型信息 Model Information
44
 
45
- 我们将[Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/)数据集(100M)和[Zero](https://zero.so.com/)数据集(23M)用作预训练的数据集,先用[IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese](https://huggingface.co/IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese)对这两个数据集的图文对相似性进行打分,取CLIP Score大于0.2的图文对作为我们的训练集。 我们使用[openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)作为初始化的text encoder,第一个stage中冻住模型的其他部分,只训练text encoder,以便保留原始模型的生成能力且实现中文概念的对齐。第二个stage中将全部模型解冻,一起训练text encoder和diffusion model,以便diffusion model更好的适配中文guidance。第一个stage我们训练了80小时,第二个stage训练了100小时,两个stage都是用了8 x A100。该版本只是一个很初步的版本,我们将持续优化模型。
46
 
47
- We use [Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/)(100M) 和 [Zero](https://zero.so.com/)(23M) as our dataset, and take the image and text pairs with CLIP Score (based on [IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese](https://huggingface.co/IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese)) greater than 0.2 as our Training set. We use [openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14) as our init text encoder. To keep the powerful generative capability of stable diffusion and align Chinese concepts with the images, We only train the text encoder and freeze other part of the model in the first stage. In the second stage, we unfreeze both the text encoder and the diffusion model, therefore the diffusion model can have a better compatibility for the Chinese language guidance. It takes 80 hours to train the first stage, 100 hours to train the second stage, both stages are based on 8 x A100. This model is a preliminary version and we will update this model continuously.
 
 
 
 
 
 
 
 
 
 
 
 
48
 
49
  ### Result
50
 
@@ -92,21 +104,6 @@ image.save("小桥.png")
92
 
93
  ## 引用 Citation
94
 
95
- 如果您在您的工作中使用了我们的模型,可以引用该模型的论文:
96
-
97
- If you are using the resource for your work, please cite the paper for this model:
98
-
99
- ```bibtex
100
- @InProceedings{Rombach_2022_CVPR,
101
- author = {Rombach, Robin and Blattmann, Andreas and Lorenz, Dominik and Esser, Patrick and Ommer, Bj\"orn},
102
- title = {High-Resolution Image Synthesis With Latent Diffusion Models},
103
- booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
104
- month = {June},
105
- year = {2022},
106
- pages = {10684-10695}
107
- }
108
- ```
109
-
110
  如果您在您的工作中使用了我们的模型,可以引用我们的[总论文](https://arxiv.org/abs/2209.02970):
111
 
112
  If you are using the resource for your work, please cite the our [paper](https://arxiv.org/abs/2209.02970):
 
42
 
43
  ## 模型信息 Model Information
44
 
45
+ 我们将[Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/)数据集(100M)和[Zero](https://zero.so.com/)数据集(23M)用作预训练的数据集,先用[IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese](https://huggingface.co/IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese)对这两个数据集的图文对相似性进行打分,取CLIP Score大于0.2的图文对作为我们的训练集。 我们使用[stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4)([论文](https://arxiv.org/abs/2112.10752))模型进行继续训练,其中训练分为两个stage
46
 
47
+ 第一个stage中冻住模型的其他部分,只训练text encoder,以便保留原始模型的生成能力且实现中文概念的对齐。
48
+
49
+ 第二个stage中将全部模型解冻,一起训练text encoder和diffusion model,以便diffusion model更好的适配中文guidance。
50
+
51
+ 第一个stage我们训练了80小时,第二个stage训练了100小时,两个stage都是用了8 x A100。该版本是一个初步的版本,我们将持续优化模型并开源,欢迎交流!
52
+
53
+ We use [Noah-Wukong](https://wukong-dataset.github.io/wukong-dataset/)(100M) 和 [Zero](https://zero.so.com/)(23M) as our dataset, and take the image and text pairs with CLIP Score (based on [IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese](https://huggingface.co/IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese)) greater than 0.2 as our Training set. We finetune the [stable-diffusion-v1-4](https://huggingface.co/CompVis/stable-diffusion-v1-4)([paper](https://arxiv.org/abs/2112.10752)) model for two stage.
54
+
55
+ Stage 1: To keep the powerful generative capability of stable diffusion and align Chinese concepts with the images, We only train the text encoder and freeze other part of the model in the first stage.
56
+
57
+ Stage 2: We unfreeze both the text encoder and the diffusion model, therefore the diffusion model can have a better compatibility for the Chinese language guidance.
58
+
59
+ It takes 80 hours to train the first stage, 100 hours to train the second stage, both stages are based on 8 x A100. This model is a preliminary version and we will update this model continuously and open sourse. Welcome to exchange!
60
 
61
  ### Result
62
 
 
104
 
105
  ## 引用 Citation
106
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
107
  如果您在您的工作中使用了我们的模型,可以引用我们的[总论文](https://arxiv.org/abs/2209.02970):
108
 
109
  If you are using the resource for your work, please cite the our [paper](https://arxiv.org/abs/2209.02970):