PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud (2024)

Abstract: Text-to-image synthesis for the Chinese language poses unique challenges due to its large vocabulary size, and intricate character relationships. While existing diffusion models have shown promise in generating images from textual descriptions, they often neglect domain-specific contexts and lack robustness in handling the Chinese language. This paper introduces PAI-Diffusion, a comprehensive framework that addresses these limitations. PAI-Diffusion incorporates both general and domain-specific Chinese diffusion models, enabling the generation of contextually relevant images. It explores the potential of using LoRA and ControlNet for fine-grained image style transfer and image editing, empowering users with enhanced control over image generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud’s Platform for AI, providing accessible and scalable solutions. All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including domainspecific ones, are publicly available. A userfriendly Chinese WebUI and the diffusers-api elastic inference toolkit, also open-sourced, further facilitate the easy deployment of PAIDiffusion models in various local and cloud environments, making it a valuable resource for Chinese text-to-image synthesis. 1

heading: 1 Introduction
text: Recently, diffusion models (Rombach et al., 2022; Saharia et al., 2022) have emerged to address the challenges of generating realistic and high-quality images from textual descriptions. This research area has obtained widespread attention, driven by the increasing demand for automated image synthesis in various applications, such as art design, virtual reality, etc (Cao et al., 2023; Sun et al., 2023).
∗Corresponding author. 1Video presentation: https://atp-modelzoo-sh.
oss-cn-shanghai.aliyuncs.com/release/ conf_demo/acl_demo_2024.mov.
In the community, Stable Diffusion2 has gained significant popularity due to its ability to generate high-quality images that align well with textual descriptions. However, when it comes to handling the Chinese language, similar models encounter certain challenges that hinder its performance. From the linguistic aspect, Chinese is a morphologically rich language with a large vocabulary size and complex inter-dependencies between characters. Furthermore, Chinese characters often have multiple meanings and can be combined to form compound words, making it challenging to establish accurate and consistent mappings between textual descriptions and visual representations (Liu et al., 2022).
Previously, in the literature, several works have been proposed to make specialized adaptations of diffusion models to improve the performance of text-to-image synthesis for Chinese (Wang et al., 2022c; Chen et al., 2023; Hu et al., 2023). Yet, there are still some notable drawbacks that need to be addressed. i) Many existing models focus on generating images based on generic textual descriptions, neglecting the ability to generate images in specific domains or contexts. ii) For the Chinese language, the potential of using LoRA (Hu et al., 2022) and ControlNet (Zhang and Agrawala, 2023) for fine-grained image style transfer and image editing have not been fully explored. iii) The lack of support for cloud-based product integration is another important drawback. Given the increasing popularity of cloud-based services and the demand for scalable and accessible text-to-image solutions, it is crucial to develop models and solutions that can be easily integrated into cloud platforms or deployed as cloud-based services.
In this work, we formally present PAI-Diffusion, which consists of a family of open Chinese diffusion models, together with user-friendly toolkits to serve these models on the cloud. Major features of PAI-Diffusion include the following:
2https://stability.ai/stablediffusion
1
• PAI-Diffusion incorporates both general and domain-specific Chinese diffusion models, specifically allowing for the generation of images that are tailored to specific contexts or domains (such as Chinese cuisine, poetry and paintings). This enables users to create visually compelling and relevant images for various domain-specific applications.
• PAI-Diffusion explores the potential of using LoRA and ControlNet for fine-grained style transfer and image editing, with a variety of corresponding Chinese models released. This empowers users with greater control over the generated images, enabling them to manipulate fine-grained semantic attributes and modify visual features based on their preferences.
• PAI-Diffusion extends our previous work (Liu et al., 2023) and ensures seamless integration with our Platform for AI (PAI) of Alibaba Cloud3, providing users with accessible and scalable solutions. By integrating with cloud products, PAI-Diffusion enables users to harness the power of cloud computing and enjoy the benefits of user-friendly and resourceefficient systems. For designers, our Chinese WebUI toolkit largely extends the Stable Diffusion WebUI4 to enrich its abilities to address the issues of the Chinese language. For application developers, our elastic inference toolkit diffusers-api makes it easy to deploy these models as RESTful web services. It further supports our compiler performance optimization tool named PAI-Blade (Zhu et al., 2021). When the functionality is enabled, the image generation speed is improved by 2-3 times compared to native PyTorch implementation, while maintaining the effectiveness.
To realize our promise for the openness of our research, we have taken the following actions to contribute PAI-Diffusion to the community:
• All the diffusion models, LoRAs and ControlNets of PAI-Diffusion, including domainspecific ones, have been released in our organizational Hugging Face repository5. Users can easily download and fine-tune all the models
3https://www.alibabacloud.com/product/ machine-learning
4https://github.com/AUTOMATIC1111/ stable-diffusion-webui
5https://huggingface.co/alibaba-pai
in order to support their own domain-specific applications.
• Our Chinese WebUI and diffusers-api toolkits have also been made publicly available6, so that PAI-Diffusion models are easy to be deployed in other environments beyond the Alibaba Cloud ecosystem, for both designers and application developers.
• All our models and codes are released under the Apache License (Version 2.0) to support both academic and commercial use.

heading: 2 Models
text: We introduce the models of PAI-Diffusion, a family of open Chinese diffusion models designed to address the challenges of generating high-quality images from Chinese textual descriptions.

heading: 2.1 Model Zoo
text: The model zoo of PAI-Diffusion consists of a collection of over ten open-source models, including base diffusion models, together with their corresponding LoRAs and ControlNets. A summary of these models is presented in Table 1, with a few cases of generated images presented in Figure 1. Readers can also find these models from our organizational Hugging Face repository described previously. Note that the list of models is not static. More new models will be added to the repository when they are ready to be released.

heading: 2.2 Model Architecture
text: To ensure full compatibility with the open-source community, our model architectures generally follow the de facto standard practice of Stable Diffusion (Rombach et al., 2022) where a CLIP-based text encoder, a U-Net and a VAE model are leveraged to obtain text embeddings, generate image embeddings in the latent diffusion space and decode the image for output, respectively. Particularly, the architectures of the U-Nets and VAEs of our large and xlarge diffusion models are in line with Stable Diffusion 1.5 and 2.1, respectively. Note that the difference between “large” and “xlarge” models is the size of generated images, rather than the number of parameters.
6The Chinese WebUI extension is released under the EasyNLP (Wang et al., 2022a,b) framework: https://github.com/alibaba/EasyNLP/ tree/master/diffusion/chinese_sd_webui The diffusers-api repository: https://github.com/ alibaba/diffusers-api
⼀位带着珠宝和皇冠标志的皇家⼥国王，8k。 A royal queen adorned with jewelry and a crown
emblem, 8k.
未来世界，机器⼈在喝酒。 A future world where robots are drinking. 在秘鲁⼯作的⼤军事⿊哈巴狗。 A military black Labrador working in Peru.
汉堡：美国汉堡。 hamburger: American hamburger.
泰迪熊在⽕星上⾏⾛，逼真，有点抽象。 A teddy bear walking on the Mars, realistic
yet slightly abstract.
⼥孩,天,阳光,微笑,光滑,⼝红,鲜花。 Girl, sky, sunshine, smile, smooth, lipstick,
flowers.
⼀群外星⼈在圆柱旁祈祷。 A group of aliens praying beside a pillar.
⻋,棋盘,⼀块,在⼀个未来烟雾缭绕的世界。 Rook, chessboard, together, in a future
world shrouded in smoke.
As for the Chinese language, we follow our previous work (Liu et al., 2023) to employ both 100 million text-image pairs from Wukong (Gu et al., 2022) and the largest Chinese KG available to us, i.e., OpenKG7 as our knowledge source to pre-train a Chinese knowledge-enhanced CLIP model that better understand the morphologically rich semantics of the Chinese language. The resources for training general-purpose models are also the same as in (Liu et al., 2023). For more details, we refer our readers to the original paper.
We have also collected a variety of domainspecific datasets to produce domain-specific diffusion models or LoRAs, depending on the volume of the corresponding datasets. Such domains include artistic pictures, paintings for Chinese poems, 2.5D-style arts, Chinese cuisines and cartoon characters. Note that we try our best to ensure that our models are built on legally and ethically sourced data. We specifically filter out any images that may have the probability to exhibit some degree of ethical bias. We obtain the permission to use in-house datasets (such as Chinese cuisine) to train and release the corresponding models.
ControlNet (Zhang and Agrawala, 2023) operates by incorporating a set of control vectors that encode specific image attributes or features. These control vectors are then incorporated into the residual blocks of the diffusion models. Furthermore, ControlNet allows for interactive image editing, enabling users to iteratively refine and modify the generated images, based on our WebUI toolkit. To enable fine-grained control, PAI-Diffusion integrates several ControlNets with Chinese diffusion models, facilitating the modification and customization of generated images based on user preferences. Currently, we have released the ControlNets based on canny edge detection (Canny, 1986) and Midas depth maps (Ranftl et al., 2022). Users can train their own ControlNets, using the same methods as ControlNet for Stable Diffusion.

heading: 2.3 Applications
text: PAI-Diffusion opens up a wide range of applications for the Chinese language. Here, we highlight some potential applications of PAI-Diffusion.

heading: 2.3.1 Artistic Creations and Design
text: Diffusion models revolutionize the way sketch images are transformed into captivating artworks.
7http://openkg.cn/
With our models, users can witness their sketch images come to life based on the image-to-image pipeline. Examples can be found in Figure 2. We can see that our models empower artists to explore their creative boundaries and produce truly unique and mesmerizing artistic creations.

heading: 2.3.2 Cultural Preservation and Heritage
text: Our models present an elegant approach to restoring ancient Chinese paintings through image inpainting techniques. By harnessing the power of diffusion, these applications enable the recreation of missing or damaged areas in the paintings, seamlessly blending them with the original artwork (with examples shown in Figure 3). Through the application, ancient Chinese paintings once damaged or fragmented can be revitalized, allowing future generations to appreciate the cultural significance of invaluable artistic treasures.

heading: 2.3.3 Visual Reality Generation
text: Diffusion models offer a cutting-edge approach to creating immersive virtual reality experiences. Take traditional Chinese garden architecture as an example (see Figure 4). Our model blends various elements such as architecture, house and furniture to generate virtual scenes reminiscent of ancient Chinese gardens. Such applications serve as a valuable tool for people to experience and appreciate the beauty of ancient Chinese culture regardless of their physical location.

heading: 2.3.4 Others
text: Apart from the three examples, our models can be harnessed in other domains such as fashion design, interior decor, and even product development. For
instance, our model can create harmonious and visually stunning environments for interior decor. In fashion design, our diffusion models can be applied to blend both traditional and contemporary styles, creating unique and captivating clothing designs. We do not further elaborate.

heading: 3 Toolkits
text: PAI-Diffusion provides user-friendly toolkits that facilitate the deployment and usage of diffusion models for Chinese text-to-image synthesis. In this section, we briefly introduce our two toolkits.

heading: 3.1 Chinese WebUI Toolkit
text: The Chinese WebUI toolkit is an extension to the Stable Diffusion WebUI specifically developed to support our Chinese models. It offers a web-based graphical interface that allows users (especially designers without programming expertise) to interact with the diffusion models easily. The toolkit provides a seamless user experience, enabling users to perform image synthesizing and editing. Snapshots of the toolkit are shown in Figure 6. Additionally, our toolkit offers two modes: single-node mode and cluster mode (shown in Figure 5), providing users with options based on their specific requirements and resources, introduced as follows:
• Single-node: It is suitable for scenarios when users want to quickly set up and use the toolkit
with exclusive computational resources. It is particularly useful for individual users or small-scale deployments.
• Cluster: Leveraging the elastic inference service of PAI, the cluster is geared towards users who require high scalability and performance. In this mode, the toolkit utilizes a cluster of nodes to handle the workload, enabling parallel processing and efficient utilization of resources. It ensures efficient scaling and responsiveness, making it suitable for enterpriselevel deployments or applications that require extensive computational power.
3.2 The diffusers-api Toolkit
The diffusers-api toolkit is built upon the Hugging Face diffusers library8 that provides a cloud service implementation for PAI-Diffusion models based on the PAI elastic inference service.9 Especially, we implement several inference pipelines customized for our Chinese models. Users can take advantage of a number of functions when sending HTTP requests, including text-to-image synthesis, imageto-image synthesis, image in-painting, image editing, etc. Below we show a sample request body for calling the diffusers-api service, with Chinese prompts manually translated into English for better understanding:
{ "task_id": "001", "prompt": "romantic starry sky", "negative_prompt": "noise, low-quality", "func_name" : "t2i", "steps": 25, "image_num": 1, "width": 512,
8https://github.com/huggingface/ diffusers
9Note that diffusers-api is also compatible with original Stable Diffusion, which is not the focus of this paper.
"height": 512, "use_base64": True
}
where “func name” refers to the specific functionality that the user wish to use (“t2i” refers to text-to-image in this case). We release the source code of diffusers-api to provide users with the ability to quickly develop customized API services. This allows for easier implementation of any desired functionality, tailored to specific requirements. For example, users can build and customize their own schedulers (Duan et al., 2023).
In addition, diffusers-api leverages the AI compiler PAI-Blade (Zhu et al., 2021) for inference optimization. It significantly reduces the end-toend latency of the inference processes and the GPU memory consumption, which ensures improved performance and efficiency in generating high-quality images, without any precision loss in computation. To verify the correctness of our argument, we have deployed our Chinese diffusion model (the large version) online using an NVIDIA A10 GPU with 50 sampling steps for inference. The generated image size is fixed to 512×512. We repeat the experiments in 20 times and report the averaged results in Table 2, which clearly prove the correctness of our claim.

heading: 4 Conclusion
text: This paper presents PAI-Diffusion to address the challenges in Chinese text-to-image synthesis. PAIDiffusion integrates both general and domainspecific Chinese diffusion models, LoRAs and ControlNets, enabling the generation of contextually relevant images. Moreover, PAI-Diffusion ensures seamless integration with our cloud platform, offering accessible and scalable solutions. The release of models further encourages collaboration and innovation in the field. The public availability of Chinese WebUI and diffusers-api toolkits simplifies deployment in various environments. Our advancements pave the way for further research and development in Chinese text-to-image synthesis and relevant applications.

heading: Acknowledgements
text: The authors would like to thank other members of the Platform for AI (PAI) team of Alibaba Cloud for the help of this work. This work was in part supported by the National Natural Science Foundation of China under grant number 62202170, Fundamental Research Funds for the Central Universities under grant number YBNLTS2023-014, Alibaba Group through Alibaba Innovative Research Program, and Alibaba Cloud through Research Talent Program with South China University of Technology.
Limitations
There are still some limitations that should be acknowledged. The effectiveness of our models heavily relies on accurate mappings between textual descriptions and visual representations. While efforts have been made to include a variety of domainspecific models, there may still be domains or contexts for which specialized models are not available. This limitation restricts the full potential in generating highly relevant and specific images for a wide range of applications. We suggest that users should further fine-tune our models if necessary.
Ethical Considerations
It is important to consider the ethical implications associated with its use and deployment. Diffusion models like PAI-Diffusion learn from large datasets, which may inadvertently contain biases present in the data. It is crucial to be aware of and mitigate any biases that may be perpetuated in the generated images. In addition, PAI-Diffusion has the potential to be misused for unethical purposes, such as generating inappropriate or harmful contents. Therefore, users should be encouraged to adhere to ethical standards and abide by terms and regulations when utilizing PAI-Diffusion models.

heading: Future Work
text: Through the application, ancient Chinese paintings once damaged or fragmented can be revitalized, allowing future generations to appreciate the cultural significance of invaluable artistic treasures