EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs (2024)

Abstract: We present EasyGen, an efficient model designed to enhance multimodal understanding and generation by harnessing the capabilities of diffusion models and large language models (LLMs). Unlike existing multimodal models that predominately depend on encoders like CLIP or ImageBind and need ample amounts of training data to bridge modalities, EasyGen leverages BiDiffuser, a bidirectional conditional diffusion model, to foster more efficient modality interactions. EasyGen achieves text generation by training a projection layer linking BiDiffuser and an LLM, and facilities image generation by training an adapter to align the LLM’s text space with the BiDiffuser’s image space. Comprehensive quantitative and qualitative experiments show that EasyGen excels in data-efficient training, high-quality image generation, and extendibility, effectively addressing the challenges in multimodal generation. The source code is available at https: //github.com/zxy556677/EasyGen.

heading: 1 Introduction
text: In recent years, remarkable progress has been made in the field of artificial intelligence generated content (AIGC), notably in technologies like large language models (LLMs) (Chiang et al., 2023; Touvron et al., 2023; Brown et al., 2020; Chowdhery et al., 2022; Zeng et al., 2022) for text generation and diffusion models (Rombach et al., 2022; Nichol et al., 2022; Saharia et al., 2022) for visual generation. These breakthroughs have paved the way for the development of multimodal large language models (MLLMs), sparking a recent trend of incorporating extra visual modules into LLMs. Collaborative models, such as Visual ChatGPT (Wu et al., 2023a) and MM-REACT (Yang et al., 2023), strategically use externally pre-trained tools to translate visual information into text descriptions and feed
*Co-second author.
the data into LLMs. However, they are exclusively dependent on pre-trained tools for inference. Contrarily, end-to-end trained models including the BLIP series (Li et al., 2023b), LLaVA series (Liu et al., 2023b,a), MiniGPT-4 (Zhu et al., 2023), and mPLUG-Owl (Ye et al., 2023) focus on mapping image information to the text space of LLMs, enabling LLMs to comprehend visual inputs.
Existing end-to-end models are also not without limitations. First, most of these multimodal models rely on either CLIP (Radford et al., 2021) or ImageBind (Girdhar et al., 2023) as their image encoder. While these encoders excel in learning unified representations that encompass both text and images, they face challenges when it comes to transforming between different modalities. This predicament makes current vision-language models relying heavily on sizable data sets to align CLIP/Bindencoded images with the language model, due to the disparity between different modalities.
Moreover, the majority of previous multimodal models have concentrated on comprehending multimodal content and lacked the capability to generate multimodal responses, such as content beyond text. Several concurrent works, such as Emu (Sun et al., 2023) and NExT-GPT (Wu et al., 2023), have utilized diffusion models for multimodal generation. Typically, these methods involve training a projection layer to align the output embedding space of the LLM with the input embedding space of the diffusion model (encoded by CLIP’s text encoder) using an MSE loss. However, this approach may lead to the underutilization of the semantic understanding and reasoning capabilities of the LLM, and may introduce information loss in the alignment process, ultimately leading to lower image generation quality compared to the original diffusion model, as elaborated in Sec. 5.6 and Tab. 6.
In this work, we propose EasyGen, an end-toend model that facilitates multimodal generation with a single bidirectional conditional diffusion
1 1351
model and LLMs, as illustrated in Figure 2. The diffusion model, called BiDiffuser, is obtained by fine-tuning the UniDiffuser (Bao et al., 2023b), with a specific focus on targeted image-to-text and text-to-image tasks. This fine-tuning addresses UniDiffuser’s limitation of attempting to fit all conditional distributions, including those based on noisy inputs, into a single model, which reduces its effectiveness on specific tasks like conditional generation from noise-free inputs. BiDiffuser plays a pivotal role for both text and image generation. In EasyGen, text generation is achieved by training a projection layer that connects BiDiffuser and an LLM, while image generation is facilitated by training an adapter that infuses the text representation of the LLM into BiDiffuser. Figure 1 showcases EasyGen’s ability to handle multimodal inputs and generate appropriate multimodal responses.
EasyGen holds three significant advantages that address the challenges in multimodal generation:
First, EasyGen offers competitive performance in a data-efficient way compared to cutting-edge models, as shown in Tab. 3 (Sec. 5.5). This is due to BiDiffuser’s ability to simplify the alignment of its embedding space with an LLM, which allows for efficient training with less data for image-to-text tasks such as image captioning and VQA.
Second, EasyGen exhibits superior image generation quality, surpassing other end-to-end MLLMs, as shown in Tab. 6 (Sec. 5.6). This is attributed to the adapter’s design (Sec. 4.2), which aligns the LLM’s text space with the diffusion model’s image space, thereby utilizing the LLM’s semantic understanding and reasoning capabilities. In contrast, the projection layers in other MLLMs like NExT-GPT only align the LLM’s text space with the diffusion model’s text space and are not trained by the image denoising objective.
Third, EasyGen can be readily adapted to manage complex vision-language tasks by incorporating more advanced visual encoders or by integrating BiDiffuser into contemporary sophisticated multimodal LLMs like LLaVA to enhance performance, as shown in Tab. 5 (Sec. 5.7).

heading: 2 Related Work
text: Multimodal Language Models. Recent research has witnessed a surge of interest in multimodal LLMs, including collaborative models (Wu et al., 2023a; Yang et al., 2023; Shen et al., 2023) and endto-end methods (Alayrac et al., 2022; Guo et al., 2022; Li et al., 2022; Bao et al., 2021; Wang et al., 2022b,a,a). More recently, some works also explore training LLMs with parameter-efficient tuning (Li et al., 2023b; Zhang et al., 2023a) and instruction tuning (Dai et al., 2023; Liu et al., 2023b; Ye et al., 2023; Zhu et al., 2023; Li et al., 2023a). Different from them, EasyGen is built upon BiDiffuser, which promotes more efficient interactions between modalities. Multimodal Diffusion Models. Diffusion generative models (Rombach et al., 2022; Ramesh et al., 2021; Nichol et al., 2022; Ruiz et al., 2023) have achieved strong results in text conditioned image
2
generation works. Specifically, Versatile Diffusion (Xu et al., 2023) employs the U-Net (Ronneberger et al., 2015) architecture with a multiflow design to tackle multiple modalities and tasks, while UniDiffuser (Bao et al., 2023b) adopts the U-ViT (Bao et al., 2023a) framework to treat both image and text as sequential token streams for diffusion calculations. However, these models are unable to complete complex language tasks. EasyGen combines the advantages of diffusion models and LLMs and achieves competitive performance in both image-to-text and text-to-image tasks.
Multimodal Response Generation. Recent research has made significant advancements in multimodal response generation (Koh et al., 2023b; Tang et al., 2023; Zhang et al., 2023b; Wu et al., 2023b; Pan et al., 2023; Koh et al., 2023a; Sun et al., 2023; Dong et al., 2023) using text-to-image models such as Stable Diffusion. However, the lack of semantic understanding capability in the CLIP text encoder may result in low-quality generated images. EasyGen addresses this issue by transferring knowledge from LLM to BiDiffuser via an adapter, enabling the creation of high-quality textual semantic representations for text-to-image generation.

heading: 3 Basics of Diffusion Models
text: Unconditional Generation. Given a data sample taken from a real data distribution x0 ∼ q(x0), diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) first destruct the data by constructing a Markov forward process and gradually injecting noise to the data:
q(x1:T |x0) = T∏
t=1
q(xt|xt−1),
q(xt|xt−1) = N (xt; √ 1− βtxt−1, βtI), (1)
where βt ∈ (0, 1) is the variance added at diffusion step t. Then, they learn to reverse the process:
p(x0:T ) = p(xT ) T∏
t=1
pθ(xt−1|xt),
pθ(xt−1|xt) = N (xt−1;µt(xt, t), σ2t I), (2)
where p(xT ) = N (xT ; 0, I) is the standard Gaussian distribution and µt(·) is the parameterization of the predicted mean. Diffusion models are trained to maximize the marginal likelihood of the data E[log pθ(x0)], and the canonical objective is the variational lower bound of log pθ(x0). Denoising
diffusion probabilistic models (Ho et al., 2020) generate samples xt ∼ q(xt|x0) by injecting noise ϵ ∼ N (0, I) to the data x0, and train a network ϵθ(·) to predict the added noise ϵ using a standard mean squared error loss:
L := Ex0,ϵ,t[∥ϵ− ϵθ(xt, t)∥2]. (3)
Conditional Generation. For conditional generation, a paired data (x0,y0) is given, and the aim is to model the conditional data distribution q(x0|y0), where y0 can be image class or text prompt. Conditional generation includes classifier guidance (Dhariwal and Nichol, 2021) and classifier-free guidance (Ho and Salimans, 2021). Classifier guidance requires training an extra classifier on noisy data at inference time to improve sample quality. For classifier-free guidance, no classifier needs to be trained. The denosing network ϵθ(xt|y0) simply conditions on the information encoded in y0. At inference time, with a guidance scale s, the modified score estimate is further in the direction of ϵθ(xt|y0) and away from the unconditional model ϵθ(xt|∅) (∅ is a null token):
ϵ̂θ(xt|y0) = ϵθ(xt|∅) + s · (ϵθ(xt|y0)− ϵθ(xt|∅)).

heading: 4 Proposed Model: EasyGen
text: We propose EasyGen, a model capable of processing multimodal inputs and generating multimodal outputs. It achieves easy multimodal generation by leveraging a bidirectional conditional diffusion model to effectively bridge the gap between different modalities and an LLM to comprehend multimodal tasks and produce textual responses containing cues for multimodal message creation. In the subsequent section, we outline the multimodal generation process of EasyGen.

heading: 4.1 Pre-training BiDiffuser: A Bidirectional
text: Conditional Diffusion Model
Since the text space of LLMs is discrete, to minimize the disparity between the output of a diffusion model and the input of LLMs, we leverage Unidiffuser, a unified diffusion model capable of transforming images into the discrete text space. During the training process, UniDiffuser injects noise ϵx and ϵy to a set of paired image-text data (x0,y0) and generates noisy data xtx and yty , where 0 ⩽ tx, ty ⩽ T represent two individual timesteps (perturbation levels). It then trains a joint denoising transformer U-ViT (Bao et al., 2023a)
3
ϵθ(xtx ,yty , t x, ty) to predict the noise ϵx and ϵy by minimizing the mean squared error loss:
Eϵx,ϵy ,x0,y0 [∥[ϵx, ϵy]− ϵθ(xtx ,yty , tx, ty)∥2],
where the output of ϵθ is the concatenation of the estimated noise ϵxθ and ϵ y θ , i.e., ϵθ = [ϵ x θ , ϵ y θ ].
By predicting ϵθ(xtx ,yty , tx, ty) for any tx and ty, UniDiffuser learns all distributions related to (x0,y0) simultaneously. This includes all conditional distributions: q(x0|y0) for text-to-image generation, q(y0|x0) for image-to-text generation, and those conditioned on noisy input, i.e., q(x0|yty) and q(y0|xtx), for 0 < tx, ty ≤ T . Learning a conditional distribution q(x0|yty) or q(y0|xtx) can be seen as learning a distinct task. From a multitask learning perspective, due to limited network capacity, learning many tasks simultaneously (i.e., fitting all distributions to a single network) may result in task competition or task conflict, ultimately leading to suboptimal performance in particular tasks such as q(x0|y0) and q(y0|x0).
To resolve this issue and enhance the performance of both image-to-text and text-to-image generation tasks, we finetune UniDiffuser with exclusive emphasis on the two tasks:
Ld = Eϵx,ϵy ,x0,y0 [∥ϵx − ϵxθ (xtx ,y0, tx, 0)∥2+ α∥ϵy − ϵyθ(x0,yty , 0, ty)∥2].
where α is a hyperparameter to balance the learning paces of the two tasks. As depicted in Figure 3, our training objective entails predicting the text y0 based on the input image x0 and vice versa, where the input conditions for the model are noisefree. We name the finetuned model “BiDiffuser”, signifying its specialized ability in bidirectional conditional generation.

heading: 4.2 Pre-training an Adapter to Enhance BiDiffuser’s SUR Capability
text: BiDiffuser uses the text encoder of CLIP, which is trained with image-text contrastive learning, limiting its semantic understanding and reasoning (SUR) ability for image generation. Drawing inspiration from Zhong et al. (2023), we utilize LLMs to enhance the SUR capability of BiDiffuser. Specifically, we design an adapter that employs the attention mechanism to integrate the semantic information from LLM’s last hidden state fLLM(·) into the CLIP text encoder fCLIP(·). The adapter consists of a projection layer MLP(·) and a crossattention layer Att(·). Given a paired image-text data (x0,y0), we can get ysur with enhanced SUR via the adapter:
ysur = Att(fCLIP(y0)WQ,MLP(fLLM(y0))WK ,
MLP(fLLM(y0))W V ).
Then, the semantic input to BiDiffuser is the combination of ysur and the CLIP text encoding of y0:
y0 = λ · ysur + (1− λ) · fCLIP(y0), (4)
where λ is a balancing parameter. We train the adapter by freezing BiDiffuser and minimizing
Lada = Eϵy ,x0 [∥ϵx − ϵxθ (xtx , y0, tx)∥2], (5)
where ϵxθ is not updated as BiDiffuser is frozen.

heading: 4.3 Image-to-Text Generation
text: BiDiffuser can convert images into vectors in the text space, facilitating alignment with the vector space of LLMs. In the following, we show how BiDiffuser can be integrated with LLMs to perform image-to-text generation tasks such as image captioning and visual question answering (VQA).

heading: 4.3.1 Aligning BiDiffuser with LLMs
text: We connect BiDiffuser and LLMs via a simple projection layer, which maps text embeddings obtained from the output of the diffusion model to the embedding space of LLMs. As shown in Figure 4, the alignment can take place either prior to the LLM (Pre-Align manner) or between its encoder and decoder components (Mid-Align manner).
Pre-Align Manner. As shown in Figure 4a, the projection layer is placed before the LLM to map the output of BiDiffuser (image representations) to the text embedding space of the LLM. The text embedding of the input image is then concatenated
4
with the embeddings of the textual instructions and fed to the LLM for decoding. To synchronize the text space of BiDiffuser with that of the LLM, we propose to use the image-grounded text generation (ITG) objective to drive the model to generate texts based on the input image by computing the autoregressive loss:
LITG = − 1
L
L∑
l=1
log pϕ(w g l |w g <l, I, TI), (6)
where wg = (wg1, ..., w g L) represents the groundtruth caption of image I with length L, TI is the text instruction, and ϕ denotes the model parameters, which include the parameters of the projection layer and the LLM.
Mid-Align Manner. As shown in Figure 4b, the projection layer is placed between the LLM’s encoder and decoder, aiming to map the output of BiDiffuser to the embedding space of the text that is encoded by the LLM’s encoder. Particularly, we argue that the output of BiDiffuser, once mapped by the projection layer and denoted as ddiff , should align with the image caption that is encoded by the LLM’s encoder, denoted as dllm. Therefore, to accurately learn the alignment between the image and text representations, in addition to the ITG loss in Eq. 6, we also employ an image-text distance minimization (ITDM) loss:
LITDM = 1
N
N∑
i=1
∥ddiff − dllm∥22,
Lmid = LITG + LITM. (7)
where N is the batch size, and Lmid is the overall loss. In this manner, the model parameters θ only include the parameters of the projection layer.
After the alignment, EasyGen gains the capability of zero-shot image-to-text generation, including tasks such as image captioning and VQA.

heading: 4.3.2 Instruction-Tuning LLMs
text: When aligning BiDiffuser with an LLM, we perform instruction-tuning on the LLM to equip it with the capability of understanding multimodal tasks. We designed different instructions for different LLMs, as shown in Table 12. General instruction template is denoted as follows: USER: <Img><image></Img> + Instruction. Assistant: <answer>.
For the <image> placeholder, we substitute it with the output of BiDiffuser. To avoid over fitting to the specific task and counter the model’s inclination to generate excessively short outputs, we have devised specific instructions (see Table 11), which enable the LLM to produce concise responses when necessary. For different tasks, the distinct instruction templates are as outlined in Appendix F.

heading: 4.4 Text-to-Image Response Generation
text: Most of existing multimodal models, including the BLIP series and LLaVA series are unable to provide a multimodal response as they are primarily designed to generate only textual outputs. On the other hand, Emu (Sun et al., 2023) takes a unified approach to predict the subsequent visual or textual token in an auto-regressive manner, but it is heavily reliant on vast quantities of training data. Contrary to the limitations of these existing models, EasyGen, by leveraging the bidirectional generation capability of BiDiffuser and the inference capability of LLMs, can produce accurate and high-quality visual response with ease.
5
To tackle multimodal response generation tasks such as PhotoChat (Zang et al., 2021), we first leverage the MLLM to generate detailed image captions based on dialogue context. Then, we employ BiDiffuser to create the corresponding images with the produced captions. Specifically, we replace the image featured in the dialogue with its corresponding descriptive caption, encapsulating it with task-specific tokens <Img>,</Img> and constructing the following instruction templates: USER: Dialog history. Assistant: <response> + <Img><caption></Img>. When <caption> appears in response, it represents the generated description of the image. So we can use LLM’s original auto-regressive training objective. Specifically, we compute the probability of the target caption by:
Lt2t = − 1
L
L∑
l=1
log pϑ(w c l |wc<l, H), (8)
where wc = (wc1, ..., w c L) represents the caption of image x0 with length L, H is the dialogue history, and ϑ denotes the LLM’s parameters. Considering the potential for alignment deviation in discrete text alone, given the description of the image y0, we utilize y0, which is the combination of the SUR adapter’s output and the CLIP text encoder’s output, as the conditional component of the diffusion model. This directly contributes to the denoising process. The loss function for the denoising process of a noisy image xtx is formulated in a way that is similar to Eq. 5:
Lt2i = Eϵy ,x0 [∥ϵx − ϵxθ (xtx , y0, tx)∥2], (9) where ϵxθ is not updated and we only train the parameters of LLM and adapter. The overall loss for text-to-image task is:
Lall = Lt2i + Lt2t. (10)
Training with the instruction data enables our model to not only produce text responses but also perform image intent classification and generate image captions that BiDiffuser can interpret.

heading: 5 Experiments

heading: 5.1 Experimental Setup
text: We initialize encoder-decoder LLM from FlanT5XL or decoder-only LLM from Vicuna-7B, along with the utilization of the diffusion module from BiDiffuser. During the alignment process, we maintain the frozen state of the BiDiffuser. The statistics of the datasets for pre-training, alignment and instruction-tuning can be found in Appendix B. For the image captioning task, EasyGen is evaluated on both the MS-COCO (Lin et al., 2014) Karpathy test set and the NoCaps (Agrawal et al., 2019) validation set. For the VQA task, we evaluated on OK-VQA (Marino et al., 2019) validation set and GQA (Hudson and Manning, 2019) test-dev set.
To adapt the model for multimodal dialogue generation, we fine-tune the LLM and projection layers on the PhotoChat dataset. We incorporate photosharing activities into the dialogue context by generating <Img><caption></Img>, and utilize crossentropy loss exclusively for fine-tuning the multimodal generation task. Given the limited expressiveness of image descriptions in the PhotoChat dataset, as evidenced by Figure 7 in Appendix I, we regenerate image annotations in a text format similar to that used in MS-COCO.

heading: 5.2 Evaluation
text: We evaluate EasyGen on various vision-language tasks including image captioning (MS-COCO (Lin et al., 2014), NoCaps (Agrawal et al., 2019)), visual question answering (OK-VQA (Marino et al., 2019), GQA (Hudson and Manning, 2019)), and multimodal dialog generation (PhotoChat (Zang et al., 2021)). We use BLIP (Li et al., 2022), Flamingo (Alayrac et al., 2022), BLIP-2 (Li et al., 2023b), InstructBlip (Dai et al., 2023), MiniGPT4 (Zhu et al., 2023), and LLaVA (Liu et al., 2023b) as baselines for image-to-text tasks, and Maria (Liang et al., 2021) and Divter (Sun et al., 2021) as baselines for the multimodal response generation task. See details in Appendix C and E.

heading: 5.3 Overall Results
text: Tab. 1 presents the evaluation results for each baseline and our models on MS-COCO and VQA (zero-
6
shot) datasets. EasyGen outperforms most of the baseline models on both the COCO test set and NoCaps validation set (zero-shot transfer). Despite being pre-trained on a small dataset (MS-COCO), EasyGen’s performance on the image captioning generation task is comparable to models trained on larger datasets. Additionally, on the OK-VQA and GQA datasets, EasyGen demonstrates improved performance compared to other models of a similar scale, achieving higher accuracy even with a simple greedy search decoding method.
In Tab. 2, the evaluation results on the PhotoChat dataset are presented. Our method exhibits clear advantages in terms of PPL, indicating strong performance on response generation task. Because of the image descriptions in the PhotoChat dataset are overly concise, we utilized EasyGen to regenerate the image descriptions, which improved the performance of our model on image generation compared to other models. Additionally, with the adapter, EasyGen is capable of generating images of superior quality.

heading: 5.4 Ablation Study
text: In Tab. 4, we examine the impact of freezing/tuning BiDiffuser and the LLM. It can be observed that frozen Mid-Align method outperforms Pre-Align method in image captioning, which shows ITDM loss function is effective. However, the frozen MidAlign method exhibits inferior performance in the VQA task. We hypothesize that this is due to the integration of mid-aligned target image features with query information, and the projection layer is insensitive to instruction information. We conduct instruction-tuning on Pre-Align T5 and Vicuna. Compared to models at the same scale, these instruction-tuned models achieve superior results.

heading: 5.5 Data Efficiency in Training
text: In Tab. 3, we examine the data efficiency of different image encoders for alignment with LLMs. EasyGen uses BiDiffuser, which maps images to the text space, simplifying alignment with LLMs. To assess the quality of visual representations, we measured the distance between the projected representations and their respective captions embedded by an LLM. We randomly selected 1,000 images with their corresponding captions from the MSCOCO dataset. The results show that our model, EasyGen, aligns significantly better with the LLM compared to other CLIP-based MLLMs, despite using less data for alignment. This indicates the effectiveness of our approach in achieving strong alignment with LLMs.

heading: 5.6 Image Generation Quality
text: Tab. 6 evaluates the generated image’s quality of MLLMs on MS-COCO validation set, using 30K randomly selected prompts to compute the FID score on generated images. To confirm the efficacy of our approach, we fine-tuned our method on a
7
portion of the original data (LIAON-COCO) and the MS-COCO train set, respectively. While other models resulted in a decrease in image generation performance compared to the corresponding diffusion model, EasyGen outperformed the original UniDiffuser due to the fine-tuned BiDiffuser and the adapter module. Furthermore, Tab. 7 provides CLIP-T scores from ImagenHub. We notice similar trends to the results in Tab. 6 using the FID indicator. This suggests that our method can better align LLM with diffusion model’s text space.

heading: 5.7 Extendability
text: Tab. 5 explores the extensibility of our method from two aspects. Firstly, we aim to enhance the performance of EasyGen on complex tasks such as VQA and OCR by integrating more powerful visual encoders. Considering the potential information dilu-
tion or omission when using BiDiffuser to convert images to text space, we choose to integrate CLIP ViT-L/14 as the image encoder (as depicted in Figure 6 in the Appendix). During this process, we freeze CLIP and BiDiffuser while fine-tuning the parameters of the LLM and projection layers. The results presented in Tab. 5 include performance on traditional short QA and the modern benchmark MMBench (Liu et al., 2023c). With CLIP ViT-L, EasyGen’s performance is better than LLaVA on the VQAv2 dataset, demonstrating that BiDiffuser can effectively assist LLM in understanding images. Secondly, we investigate the plug-and-play capability of BiDiffuser, as it can also be integrated into other MLLMs (with the same LLMs) to improve their performance. The results demonstrate that with BiDiffuser, LLaVA-1.5 could achieve better performance. We speculate that BiDiffuser provides guidance information to MLLMs, enabling them to better understand the details of CLIP encoded images.
8

heading: 6 Conclusion and Future Work
text: We have introduced EasyGen, a model that facilitates multimodal understanding and generation. Compared to existing models, EasyGen offers a more efficient solution by employing BiDiffuser, a bidirectional diffusion model. This allows for more effective modal interactions, handling both imageto-text and text-to-image generations by the fusion of BiDiffuser and LLMs. Additionally, EasyGen can be easily integrated into existing advanced multimodal LLMs to further boost their performance. In the future, we will explore adapting EasyGen to perform a broader range of multimodal tasks, including subject-driven image generation, image editing, and controlled generation.

heading: 7 Limitations
text: This section aims to highlight the limitations of our work and provide further insights into the research in this area. Our model relies on diffusion for multimodal interaction, which means that the text-toimage and image-to-text processes may take longer. In our experiments, we tested the performance of our model on one A100 (80G) GPU. During inference, using 1000 image-caption pairs, EasyGen took approximately 2.95 seconds for the caption generation task (with the diffusion module taking about 2.41 seconds) and around 4.96 seconds to generate an image. We believe it would be beneficial to explore more efficient sampling methods, such as DPM-Solver++ (Lu et al., 2022), to improve the overall efficiency of EasyGen. Furthermore, EasyGen may not be seamlessly adaptable for jointly fine-tuning the BiDiffuser and Language Model without altering the BiDiffuser’s sampling mechanism. But based on our experimental findings, that joint fine-tuning of the BiDiffuser and LLM is not necessary for handling complex tasks. This observation aligns with many established approaches that do not require fine-tuning of their corresponding image encoders.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable feedback. This research was partially supported by the grant of HK ITF ITS/359/21FP.

heading: A Ethics Statement
text: We adhere to the ACL Ethics Policy and have conducted our research using publicly available repositories and datasets. Our primary focus is on investigating the integration of diffusion models and LLMs for multimodal generation. Therefore, the results should be seen as AI-generated content. While we have not observed deliberate harmful content, the model has the potential to generate such content if triggered. We have taken steps to minimize this risk through fine-tuning on public datasets, but caution is still exercised. In future, we will prioritize improving downstream performance and exploring methods to enhance control over the generation process. To ensure reproducibility and support future research, we have made all resources publicly available and provided proper citations to previous research within the code.

heading: B Datasets
text: We test the effectiveness of EasyGen by experimenting on different tasks including image captioning, visual question answering (VQA), and multimodal dialogue tasks. Table 8 shows the statistics of the pre-training datasets for BiDiffuser, alignment and VQA tasks.
We use the MS-COCO (Lin et al., 2014) dataset for image captioning. Following BLIP-2 (Li et al., 2023b), we fine-tune EasyGen on MS-COCO and evaluate its performance on the Karpathy test set and the NoCaps (Agrawal et al., 2019) validation set. In MS-COCO, each image typically has five captions that convey similar meanings. The training set consists of 82,783 images with 414,113 captions, while the COCO Karpathy test set has 5,000 images and the NoCaps validation set has 4,500 images.
For multimodal dialogue, we utilize the PhotoChat (Zang et al., 2021) dataset, which is a highquality dataset consisting of 10,917 images and 12,286 dialogues. Each dialogue is associated with a user image and its corresponding text description. The dataset is divided into 10,286 training instances, 1,000 development instances, and 1,000 testing instances. Moreover, PhotoChat includes photo-sharing activities, defined as the process of creating <Img><caption></Img> in this study. Each conversation in PhotoChat is broken down and constructed into multiple samples so that each round of responses can be learned. Specifically, we regard the first three turns as the dialog context,
and the subsequent turns as the prediction targets. By converting the dialogues of this dataset into the form mentioned in 4.4, we obtained 49,240 train, 4,792 dev, and 4,836 test dialogue pairs.
For the VQA task, we conduct a quantitative evaluation on both the OK-VQA (Marino et al., 2019) validation set (5,046 questions) and the GQA (Hudson and Manning, 2019) test-dev set (12,578 questions). As shown in Table 4, for the frozen LLM, following BLIP-2, we employ the length penalty in beam search to encourage short answer generation. On the contrary, for the tuned LLM, we use the VQA instructions (as shown in Table 10) to do instruction tuning during the alignment process. The data for instruction tuning is constructed by randomly selecting 5K data from VQAv2 (Goyal et al., 2017) and 5K data from Visual Dialog (Murahari et al., 2019) training set.

heading: C Baselines
text: We compare our proposed model with the following state-of-the-art baselines: BLIP (Li et al., 2022) is a multimodal mixture of encoder-decoder. It can be used as an imagebased text encoder or decoder. We use it to perform caption generation and VQA tasks. BLIP-2 (Li et al., 2023b) is pre-trained through bootstrapped learning from frozen visual encoder and LLMs using an efficient pre-training strategy. Flamingo (Alayrac et al., 2022) incorporates new cross-attention layers into Chinchilla language model (Hoffmann et al., 2022) to inject visual features, and pre-trains the new layers on billions of image-text pairs. We use it to perform caption generation and VQA tasks. InstructBlip (Dai et al., 2023) is a vision-language instruction tuning framework that is trained with BLIP-2 and capable of solving various visual language tasks. MiniGPT-4 (Zhu et al., 2023) utilizes a single projection layer to align visual information from a pretrained vision encoder with an LLM. It employed the same visual encoder as used in BLIP-2. LLaVA (Liu et al., 2023b) employs a solitary projection layer to convert image features extracted from the pre-trained CLIP-ViT-L/14 visual encoder into the language embedding space of Vicuna. Maria (Liang et al., 2021) is a neural conversation agent which can leverage visual world experiences sourced from a vast image index. It possesses the ability to fetch a relevant image specific to the con-
13
versation and extract visual knowledge from it. Divter (Sun et al., 2021) focuses on exploring multimodal dialogue generative models. Given the dialogue context, this model first generates a text response or image description and then generates an image according to the description.

heading: D Evaluation
text: For evaluating the quality of text generation, we utilize metrics such as BLEU, Rouge-L, Accuracy, and PPL (Perplexity). Additionally, following the approach of Vicuna (Chiang et al., 2023) and LLaVA (Liu et al., 2023b), we employ ChatGPT to assess the generated responses from our model. Specifically, for the image captioning task,
we randomly select 30 images from the MS-COCO Karpathy split and then let ChatGPT score the responses generated by EasyGen and the baseline models. ChatGPT evaluates the models’ responses based on relevance, details, and accuracy and assigns an overall score between 1 and 10, with a higher score indicating better performance. To evaluate the quality of image generation, we use the Frechet Inception Distance (FID) score (Heusel et al., 2017), which measures the divergence between two multivariate normal distributions.
E Implementation Details
LLM During the alignment process, we utilize the AdamW optimizer with β0 = 0.9, β1 = 0.99, and weight decay of 0. The LLMs are trained with a cosine learning rate of 2e-5 and a warmup rate of 0.03. We use a batch size of 96 for the frozen LLMs and 32 for the tuned LLMs. During training, we convert the LLMs (FlanT5XL/Vicuna-7B) to BFloat16/FP16 and BiDiffuser to FP16. During the VQA tuning process, we use CLIP ViT-L/14 336px as additional image encoder. We finetune EasyGen on mixture datasets for 1 epoch with a batch size of 32. We adopt the AdamW optimizer with β = (0.9, 0.99) with the learning rate is 2e-5. We use a cosine learning rate decay with a learning rate is 2e-5 and warmup ration is 0.03.
Diffusion Module We inherit the settings from UniDiffuser and utilize pre-trained weights from its checkpoint for our text-to-image generator. The model is fine-tuned on the MS-COCO and VG
14
dataset, which contains images with a resolution of 512 × 512, for 10 epochs with a batch size of 312. For all of our sampling processes, we employ DPM-Solver with 50 steps.
F Instruction Tuning
We list the instructions for different tasks in the main paper in Table 10. Specifically, the queries used to describe image contents are presented in Table 11. Table 10 shows the templates used in Vicuna, if the LLM is FlanT5, kindly use “Human” to substitute “USER” in the instruction templates. Model architecture for VQA finetuning is shown in Figure 6. EasyGen integrates the outputs of BiDiffuser with images encoded by CLIP ViT-L/14. We freeze CLIP and BiDiffuser while only tuning the parameters of the LLM and projection layers.

heading: G Training Efficiency
text: Table 13 summarizes the key factors in training EasyGen. The training process of EasyGen is computationally efficient, especially with the utilization of the parameter-efficient fine-tuning method LoRa (Hu et al., 2021). To enable multimodal response generation, we further train the aligned EasyGen. This process entails fine-tuning the LLM (FlanT5XL) on the PhotoChat dataset for 2 epochs, which typically requires approximately 4 A100 (80G) GPU hours.
H Impact of Alignment Manners
In Table 14, we investigate the impact of different alignment manners on EasyGen. After removing the ITDM loss, the performance is slightly weaker than the original model. It is evident that the MSE Loss can help to align the semantic spaces of the two models. Furthermore, the performance of the model will drop significantly after removing the cross-entropy loss, suggesting that constraints via the language model play a key role.

heading: I More Qualitative Results
text: We present several instances on PhotoChat dataset in Figure 7 and the image-captioning task in Figure 8. In Figure 9, 11, 10, we compare EasyGen with state-of-the-art multimodal language models. The responses of MiniGPT-4, LLaVA, mPLUGowl and InstructBlip are obtained from their official demos. Morever, in Figure 12, 13, we show EasyGen’s ability to accept multimodal inputs and generate multimodal responses.
15
16
MultiModal Chat Group
Please tell me the information about this picture.
Generating some pictures like this.
This painting is a classic representation of a young girl from the 17th century. The girl is depicted wearing a blue and white dress, with a pearl earring in her ear. The painting is a masterpiece of Dutch Golden Age art, a period of great cultural and economic prosperity in the Netherlands. The girl in the painting is likely depicted as a middleclass citizen, and the painting may have been commissioned as a portrait or for a specific purpose, such as a family crest or a religious painting. The painting is known for its attention to detail, vivid colors, and beautifully rendered textures. It is a true masterpiece of Dutch art, and a must- see for anyone interested in the history of art.
EasyGen (ours):
EasyGen (ours):
mPLUG-Owl:
mPLUG-Owl:
LLaVA:
LLaVA:
MiniGPT-4:
MiniGPT-4:
The image is a painting depicting a young girl, likely a Dutch girl in a yellow dress, with her hair tied up in braids and a gold earring. She is looking directly into the viewer’s eyes, creating an intimate connection with the observer. The painting is set in an indoor scene with a table and chairs visible in the background. The girl is positioned in front of the table, and there is an additional person standing to her left. There are also several other people in various positions around the room, suggesting a social gathering or a family setting.
Here are a few examples of paintings that are similar in style to the one you described: 1. “Girl with a Pearl Earring” by Johannes Vermeer; 2. “The Procuress” by Pieter de Hooch; 3. “The Artist’s Mother” by Gerard van Honthorst …. These paintings are all from the 17th century and depict women or groups of people in various settings, often with a focus on everyday life. They are known for their attention to detail and realistic depiction of light and shadow.
The image is a portrait of a woman wearing a blue headscarf, a yellow dress, and a pearl earring. She is looking to her left, and her outfit and accessories create a visually appealing contrast against the background. The woman’s attire and appearance suggest a sense of style and elegance.
I’m sorry, as an AI language model, I am not able to generate images. However, I can suggest some ideas for paintings that share a similar style to the portrait of the woman in the blue headscarf, yellow dress, and pearl earring by Johannes Vermeer:
The painting depicts a young woman with a pearl earring looking out into the distance. The woman is wearing a yellow dress with a blue and white headband. The background is a mix of blue and green, with a few trees visible in the distance. The overall mood of the painting is peaceful and contemplative. The painting is likely a representation of a woman reflecting on her thoughts or feelings.
17
MultiModal Chat Group
Where can I see this object appeared in this image?
I plan to travel here, what should I pay attention to?
EasyGen (ours):
mPLUG-Owl:
LLaVA:

heading: As you’re looking at a rendering of an iconic white and gold statue of a woman in a field, pay attention to the details of the sculpture, the sky, and the surroundings. The statue is situated in a green field, which provides a picturesque and serene atmosphere. The sky in the background is blue, suggesting a clear and sunny day.

heading: To fully appreciate the beauty and significance of the statue, it’s crucial to observe its detail, size, and the overall composition of the image. The statue may depict a woman, an important figure in history, or a symbol of liberty and freedom. Additionally, pay attention to the field’s surroundings, such as the grass,
text: vegetation, and any other elements that contribute to the overall aesthetic.
When visiting the Statue of Liberty, there are a few things you should keep in mind:

heading: 1. The Statue of Liberty is a colossal neoclassical sculpture on Liberty Island in New York Harbor within New York City, United States. It was a gift from the people of France to the people of the United States and was dedicated on October 28,

heading: 1886, to commemorate the centennial of the signing of the United States Declaration of Independence.

heading: 2. The statue is 301 feet (92 meters) tall, including its pedestal and base, and is made of iron and steel. It was designed by the French sculptor Frederic Auguste Bartholdi and the American architect Emmanuel Louis Masqueray.
text: 3. To get to the Statue of Liberty, you can take a ferry from Battery Park in Lower Manhattan. There are several ferry services that offer tours to the statue, and it’s important to plan ahead and book your tickets in advance to avoid long lines. Once you’re on Liberty Island, you can take an elevator or stairs to the top of the pedestal, where you’ll have a stunning view of

heading: 4. New York Harbor and the surrounding city skyline. It’s important to note that the pedestal is closed to visitors, but you can still get a close-up view of the statue from the observation deck.

heading: 5. Be sure to check the weather forecast before your visit, as the statue can be closed due to high winds or other inclement weather conditions. And don’t forget to bring your camera to capture the stunning views and memorable moments!

heading: 1. Tickets: You will need to purchase tickets in advance to visit the Statue of Liberty and Ellis Island. You can do this through the official Statue of Liberty National Monument website or through authorized ticket sellers.

heading: 2. Accessibility: The Statue of Liberty is accessible via ferry from Battery Park in Lower Manhattan. There are also guided tours available, which can provide more information about the history and significance of the statue.

heading: 3. Closed areas: Some areas of the Statue of Liberty, such as the pedestal and the crown, are closed to visitors due to safety concerns. Make sure to check the official website for the most up-to-date information on accessibility and restrictions.

heading: 4. Clothing and footwear: Wear comfortable shoes and dress appropriately for the weather, as you will be
text: The Statue of Liberty is located in New York City, United States. It is a famous landmark and tourist attraction, and you can see it in person by visiting Liberty Island in New York Harbor.
EasyGen (ours):
mPLUG-Owl:
LLaVA:
You can see this blue and white 3D rendering of the Statue of Liberty located in the United States. The image features an artistically detailed representation of the famous landmark, which stands tall and proud in the skyline. The statue represents the ideals of freedom, democracy, and human rights. The scene is captured in a blue sky, emphasizing the beauty and grandeur of the iconic structure.
You can see the Statue of Liberty in the image, standing tall and proud in front of a blue sky.
18
How can the girl improve her working environment?

heading: In the image, the young girl is sitting in a room with a laptop in front of her, which suggests that she might be working or studying. To improve her working environment, she can consider several factors:

heading: 1. Ergonomics: Ensure that the laptop is placed at eye level or slightly below, and the desk is at a comfortable height for her to maintain a neutral posture. Adjustable furniture like a desk or a chair can help achieve a comfortable working height.

heading: 2. Lighting: Ensure that the room is well-lit, with natural light or artificial lighting to minimize eye strain and improve focus.

heading: 3. Organization: Keep the room clean and organized, with important items within easy reach. This can help maintain focus and prevent distractions.

heading: 4. Comfortable environment: Creating a comfortable environment, such as a dedicated workspace separate from her bedroom, can help establish boundaries and promote productivity.

heading: 5. Time management: Establish a routine and allocate specific time slots for studying or working on her laptop. This can help her stay on track and avoid procrastination.

heading: By implementing these suggestions, the girl can enhance her working environment and achieve
text: better focus, productivity, and overall comfort while working on her laptop.
EasyGen (ours):
mPLUG-Owl:
MiniGPT-4:

heading: To improve her working environment, the girl could consider several options:

heading: 1. Organize her space: Ensure the area where she is working is clean, uncluttered, and well-lit. Remove distractions such as toys or other items that might cause her to lose focus. Arrange her books and laptop in a manner that is easy to access and comfortable for her to work in.

heading: 2. Create a designated work area: Set up a designated space for her to work, such as a desk, table, or a specific corner in the living room. This helps her maintain her focus and minimize distractions.

heading: 3. Provide a comfortable chair and proper lighting: Select a chair with good back support and proper ergonomics to ensure she is comfortable while working. Lighting should be bright and evenly distributed to minimize shadows and glare.

heading: 4. Set reasonable work hours: Set a reasonable time limit for her to work, and let her know when it’s time to take a break. This can help her maintain focus and minimize fatigue.

heading: By creating a well-organized, comfortable, and productive working environment, the girl can improve her focus and efficiency while minimizing distractions during her time spent studying and doing homework.
text: There are several ways the girl can improve her working environment. One way is to make sure her desk and chair are comfortable and adjustable to her height. Another way is to have a clean and organized workspace, so she can easily find what she needs. Additionally, she could have a good lighting system to help her see her work better. Finally, she could have a cushion or pillow to sit on to help reduce fatigue.
InstructBlip:
MultiModal Chat Group
19
20

heading: Future Work
text: In future, we will prioritize improving downstream performance and exploring methods to enhance control over the generation process. To ensure reproducibility and support future research, we have made all resources publicly available and provided proper citations to previous research within the code