Taiyi-Diffusion-XL: Advancing Bilingual Text-to-Image Generation with
  Large Vision-Language Model Support
Authors: name: Xiaojun Wu; name: Dixiang Zhang; name: Ruyi Gan; name: Junyu Lu; name: Ziwei Wu; name: Renliang Sun; name: Jiaxing Zhang; name: Pingjian Zhang; name: Yan Song
Published: 2024-01-26 07:17:50

Abstract: Recent advancements in text-to-image models have significantly enhanced image
generation capabilities, yet a notable gap of open-source models persists in
bilingual or Chinese language support. To address this need, we present
Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model
which is developed by extending the capabilities of CLIP and
Stable-Diffusion-XL through a process of bilingual continuous pre-training.
This approach includes the efficient expansion of vocabulary by integrating the
most frequently used Chinese characters into CLIP's tokenizer and embedding
layers, coupled with an absolute position encoding expansion. Additionally, we
enrich text prompts by large vision-language model, leading to better images
captions and possess higher visual quality. These enhancements are subsequently
applied to downstream text-to-image models. Our empirical results indicate that
the developed CLIP model excels in bilingual image-text retrieval.Furthermore,
the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass
previous models. This research leads to the development and open-sourcing of
the Taiyi-Diffusion-XL model, representing a notable advancement in the field
of image generation, particularly for Chinese language applications. This
contribution is a step forward in addressing the need for more diverse language
support in multimodal research. The model and demonstration are made publicly
available at
https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/{this
https URL}, fostering further research and collaboration in this domain.