Decoder-Only LLMs are Better Controllers for Diffusion Models

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 08:01:58

Abstract: Groundbreaking advancements in text-to-image generation have recently been achieved with the emergence of diffusion models. These models exhibit a remarkable ability to generate highly artistic and intricately detailed images based on textual prompts. However, obtaining desired generation outcomes often necessitates repetitive trials of manipulating text prompts just like casting spells on a magic mirror, and the reason behind that is the limited capability of semantic understanding inherent in current image generation models. Specifically, existing diffusion models encode the input text prompt with a pre-trained encoder structure, which is usually trained on a limited amount of image-caption pairs. State-of-the-art large language models (LLMs) based on the decoder-only structure have shown very powerful semantic understanding capability as their architectures are more suitable for training on very large-scale unlabeled data. In this work, we propose to enhance text-to-image diffusion models by borrowing the strength of semantic understanding from large language models (LLMs), resulting in a simple yet effective adapter to allow the diffusion models to be compatible with the decoder-only structure. In the evaluation, we conduct not only extensive empirical results but also the supporting theoretical analysis with various architectures (e.g., encoder-only, encoder-decoder, and decoder-only). The experimental results show that the enhanced models with our adapter module are superior to the stat-of-the-art models in terms of text-to-image generation quality and reliability.

Main paper: 
 introduction: 
 Image generative models have progressed explosively in recent years, with the prevalence of Generative Adversarial Networks (GANs) and diffusion models. Text-to-image generation methods such as Stable Diffusion [24, 27] , DALL-E 3 [2] , and Imagen [28] are capable of synthesizing high-quality images by taking textual descriptions (prompts) as the input. One key step of these models is to understand the user intention and semantic meanings from the text prompts and encode them to text features for further driving image content generation with diffusion models. To this end, most of the existing methods adopt an encoder-based language model structure (e.g., CLIP [25] or T5 [26] ), which were pre-trained on limited amount of image-caption pairs or texts pairs due to the expensive data annotation cost, resulted in the unsatisfying performance for the image generation quality and reliability. As a result, obtaining a user-desired image with these methods is very hard especially with a purpose of generating a complex and detail-rich image, and repetitive trials on manipulating the text prompts are nearly a must have. For example, as shown in Fig. 1 , the state-of-the-art DALL-EFigure 2 : Comparison with other neural network structures employed for computing text encoding in diffusion models. Our proposed LLMDiff, which leverages a decoder-only structure with casting the transformer-based language model as a diffusion model, can predict the text encodings for text-to-image generation by integrating layer-wise representations in the language model. Intuitively, compared with other structures (e.g. encoder-decoder) our LLMDiff is more powerful on exploring the semantic meanings and dependency among words from the input text prompt. Please refer more details and theoretical derivations in Sec. 3.4.3 fails to comprehend the entities and their relationships described in the complex prompts, resulting in numerous omissions.On the other hand, we have also witnessed a very fast development on the Large Language Models (LLMs), e.g., GPT-4 [22] , PaLM [7] and Llama2 [14] , which have show very incredible power on semantic understanding, reasoning and naturally interacting with human. These LLMs mainly employ the decoder-only structure that can be trained on a massive scale of unlabeled textual data. Unfortunately, bridging the ability of LLMs with current diffusionbased text-to-image generation framework is unexplored due to the incompatibility of these two model architectures. Some recent attempts have made on borrowing the ability of LLMs for enhancing the text-to-image generation performance with the diffusion models [11, 18] . Their approaches proposed to enrich or rewrite the user text prompt through LLMs and still rely on the vanilla text encoders to guide the image generation process within the diffusion models, leading to sub-optimal performance.To tackle this challenge, we propose a novel and general approach to upgrading various text-to-image diffusion models by borrowing the strength of semantic understanding from large language models (LLMs). In particular, we reveal that a Transformer-based language model (e.g. ChatGPT [22] ) can be rephrased as the denoising steps in Denoising Diffusion Probabilistic Models (DDPMs) [12] . Viewing LLMs as diffusion models, we have further derived theoretical underpinnings for extracting text encodings from the blocks of LLMs. These findings drive us to attach a simple yet effective network module to the cross-attention part of the denoising U-Net. as shown in Fig. 2 . And this module enable us to effectively integrate block-wise representations within the language model for generating the text encoding of the input text prompt, which can accurately capture the semantic meanings and contextual dependency among words due to the power of pre-trained LLMs. We name this module as LLMDiff-Adapter as it can be very compatible plug and play component for connecting LLMs with various textto-image diffusion models and gaining conspicuous improvement.As some examples shown in Fig. 1 , the results generated by our model can better preserve the semantic meanings and user intent from the input prompts, e.g. well representing the entities and their relationships for the image generation.In the evaluation, we conduct a comparative analysis on different text-to-image models on the same benchmarks. We compared the performance of using our proposed LLMDiff-Adapter against other architectures, e.g. simply connecting the output of decoder-only LLMs or adopting encoder structure such as CLIP [25] and T5 [26] through linear layers to the diffusion models. The experimental results show that our model achieve superior performance among the competitions in several aspects, including the quality of generated image details, logical coherence, and comprehensive understanding of the text descriptions. The relevant quantitative results are also presented for underscoring the effectiveness of our approach in solving the limitations of current diffusion-based text-to-image generation methods. 
related works 2.1 text-to-image diffusion models: 
 Recently, diffusion-based image generation models have achieved remarkable success. These models learn to iteratively denoise a noisy image and generate the image progressively [12] . Compared to GAN-based methods, diffusion models are more stable in training and able to generate more diverse images. With the advent of diffusion models incorporating guidance mechanisms [13, 19] , there has been a notable advancement in the performance of diffusion models. For the first time, diffusion models beat GANs in conditional generation tasks. Ever since, the focus of research on text-to-image synthesis has gradually shifted from GAN to Diffusion [5, 15, 16, 21] . Some large-scale text-to-image models [2, 10, 27, 31] have achieved highly accurate and fine-grained controllable semantic generation.The recently proposed latent diffusion model (LDM) [27] unprecedentedly makes high-resolution and high-quality text-to-image models become a reality. Based on LDM, DALL-E 3 [2] has ushered the text-to-image models into unprecedented levels, leveraging powerful text encoders and high-quality data.In recent years, large-scale language models with billions of parameters have demonstrated remarkable performance across various natural language understanding and generation tasks. The dominant form of language models shifted from BERT-like models [9, 20] that focus on language understanding to the currently prevalent generative language models with decoder-only architectures [7, 14, 17, 22] . These decoder-only models have successfully unified a wide spectrum of tasks, showing commendable proficiency in dialogue interactions. Even in language comprehension tasks, [4, 8] also shows that CLIP and BERT style text encoders perform worse than decoder-only LLMs. Moreover, recent models exhibit the ability of in-context learning [3] , enabling them to adaptively leverage contextual information to accomplish downstream tasks.Existing text-to-image diffusion models are primarily based on encoder-structured text models like CLIP and T5. However, there are ongoing efforts of works that seek to explore the potential for transposing the wealth of knowledge inherent in Large Language Models (LLMs) into existing diffusion frameworks. Certain research endeavors, such as [11, 18] , have attempted to utilize LLMs to predict the layout of objects, thereby enhancing the logical coherence and overall quality of the images produced by diffusion models. This is achieved by employing LLMs to rewrite the prompts, ensuring a better alignment between the generated images and the input text. Another approach [1] attempts to use LLMs to help users construct better prompts, leveraging the capabilities of LLMs to generate superior images.While these pioneering efforts are indeed instrumental in integrating the knowledge of LLMs into diffusion models, they predominantly employ indirect methods to bridge the gap between them, and thus, are inherently constrained by the limitations of the inefficient text encoder. In contrast, we propose a novel method that directly integrates the output of the LLM into the existing diffusion model. By completely discarding the text encoder, we aim to liberate the text-to-image diffusion models from the bottleneck of language comprehensibility, which may significantly enhance their performance in controllable image generation. 
text-to-image diffusion models: 
 Text-to-image diffusion models typically employ an encoder to encode textual inputs 𝑥 with 𝑑-1 tokens as control conditions 𝑐 <𝑑 . Sequentially, those text encodings 𝑐 <𝑑 are decoded for image generation through the diffusion model, i.e., 𝑝 (𝑧 𝑡 -1 |𝑧 𝑡 , 𝑐 <𝑑 ), where 𝑧 𝑡 is the latent at timestep 𝑡, or for text generation via a text decoder, i.e., 𝑝 (𝑥 𝑑 |𝑐 <𝑑 ), where 𝑥 𝑑 is the 𝑑-th predicted token.Typically, the text encoder utilized by diffusion models is derived from pre-trained models such as encoder-only or encoder-decoder LLMs. However, despite their impressive generative performance, decoder-only LLMs are not applicable to text-to-image generation. This is because these models directly generate tokens, making it infeasible to get text features 𝑐 directly.We revisit the transformer-based LLMs from a probabilistic perspective, to help to derive the formal modeling of text encodings for text-to-image generation. Considering that LLMs in a transformer architecture have a sequence of transformer blocks with the same structure, it is intuitive to model the forward process in a diffusionlike manner. Take an encoder-only LLM, CLIP, for example. Each input token is first fed into an embedding layer. For similarity, the output of the embedding layer for the 𝑑-th token is taken as the input, denoted as 𝑥 𝑇 𝑑 . Then, it goes through 𝑇 transformer blocks that perform the causal attention masks in self-attention, which can be represented as 𝑝 𝜃 𝑡 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 , 𝑥 𝑡 <𝑑 ) for the 𝑡-th block parameterized 𝜃 𝑡 . This process is akin to the denoising process of DDPM with conditioning. So, the transformer-based LLMs can be viewed as diffusion models. Thus, we can leverage the dynamical properties and theoretical frameworks of diffusion models to analyze various structures of LLMs with a causal mask.Moreover, the prediction of the model can be formulated as:EQUATIONFor an encoder-decoder LLM, the encoder model processes contextual text, encoding it into a feature representation, i.e., text encodings 𝑐 <𝑑 . Subsequently, the decoder model utilizes these text features to generate words with 𝑝 𝜃 𝑡 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 , 𝑐 <𝑑 ). Thus, each block in the decoder utilizes the same condition 𝑐 <𝑑 . Using the input 𝑥 𝑡 𝑑 and output 𝑥 𝑡 -1 𝑑 of any block, the encoding 𝑐 <𝑑 from the encoder is estimated through Bayes' theorem: token based on conditions from the preceding tokens. Those contextual conditions are changing, not shared like encoder-decoder LLMs. Namely, when predicting the 𝑑-th word, the preceding 𝑑-1 words collectively serve as its contextual condition,EQUATIONEQUATIONAccordingly, given the input 𝑥 𝑡 𝑑 and output 𝑥 𝑡 -1of transformer blocks, the estimation of 𝑝 (𝑥 𝑡 <𝑑 |𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 𝑑 ) can be derived as follows,𝑝 𝜃 𝑡 (𝑥 𝑡 <𝑑 |𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 𝑑 ) = 𝑝 (𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 𝑑 |𝑥 𝑡 <𝑑 )𝑝 (𝑥 𝑡 <𝑑 ) 𝑝 (𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 𝑑 ) = 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 ≤𝑑 )𝑝 (𝑥 𝑡 𝑑 |𝑥 𝑡 <𝑑 )𝑝 (𝑥 𝑡 <𝑑 ) 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 )𝑝 (𝑥 𝑡 𝑑 ) = 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 ≤𝑑 )𝑝 (𝑥 𝑡 <𝑑 |𝑥 𝑡 𝑑 ) 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 ) ∝ 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 , 𝑥 𝑡 <𝑑 ) block prediction of sentence / 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 )block prediction of single words (4) where 𝑝 (𝑥 EQUATIONwhere 𝑔(𝑡) is a scalar function that is dependent on the time step 𝑡. Furthermore, from Eqn. (1) , by modeling an LLM as a diffusion process, the score function for 𝑝 (𝑐 <𝑑 |𝑥) can be approximated as:EQUATIONTaking into account the stochastic nature of generative language models during sampling, we can use this score function to perform Langevin dynamics sampling to obtain the final text encoding for image generation:EQUATIONwhere ℎ(𝑡) is a learnable function, and 𝜖 𝑡 ∼ N (0, 𝐼 ). 
llmdiff adapter 4.1 decoder-only llms as diffusion controller: 
 As discussed in Sec. EQUATION)Leveraging the residual structure of existing transformer blocks and by combining Eqns. ( 5) and ( 6), we can transform these transformer blocks to derive the model for predicting scores:𝑆 𝜃 𝑡 (𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 ≤𝑑 ) ≈ ∇ 𝑥 log 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 , 𝑥 𝑡 <𝑑 ), 𝑆 𝜃 𝑡 (𝑥 𝑡 -1 𝑑 , 𝑥 𝑡 𝑑 ) ≈ ∇ 𝑥 log 𝑝 (𝑥 𝑡 -1 𝑑 |𝑥 𝑡 𝑑 ).Accordingly, the estimation of 𝑐 is implemented by Algorithm 1. Based on this text encoding, we can construct an adapter to integrate decoder-only LLMs into existing diffusion models. In contrast to the primary practice of merely employing LLMs for text optimization and subsequently encoding texts via a text encoder with inherent performance limitations, text encodings derived from LLMs to control the generation process of diffusion models can be a superior alternative for diffusion model training from scratch or adaption in a pre-trained diffusion model. In the following, we will elaborate an effective adaptor in a pre-trained diffusion model for image generation.To leverage the pre-trained knowledge of existing diffusion models more effectively, we propose an LLMDiff Adapter incorporating text encoding from generative decoder-only LLMs into a pre-trained text-to-image diffusion model, as illustrated in Fig. 3 . The original cross-attention module is aligned with the preceding text encoder, and it is what actually imposes a bottleneck on the comprehension of user prompts. However, it still holds a wealth of knowledge and insights for text-to-image generation, learned during the pretraining phase. Therefore, we keep the original cross-attention module intact and align it with the encoding derived from LLMs through linear layers. This enables effective utilization of the knowledge of large-scale pre-trained models, preserving basic generation capabilities.Simultaneously, an additional cross-attention module is introduced to learn how to better generate images based on the text encoding derived from LLMs. The outputs of these two modules are combined through a set of learnable weight factors: 𝑎 1 , 𝑎 2 , 𝑏 1 , and 𝑏 2 , and the overall computation can be formulated as follows:EQUATION)where τ is the linear layer of the original cross-attention module, 𝜏 is that in additional cross-attention module, and 𝜙 is the linear layer to align the LLMs with the original cross-attention module.For training stability, the initial values of 𝑎 1 and 𝑏 1 are set to 1 and 0, respectively, while 𝑎 2 and 𝑏 2 start at 0. During the model learning, the newly added cross-attention module gradually refines the outputs, effectively adapting the knowledge of generative LLMs to the diffusion model. Our Adapter is trained with the MSE loss for diffusion models:EQUATION)where 𝜖 𝜃 is the diffusion U-Net, 𝑧 𝑡 is the latent feature map at timestep 𝑡, and 𝜖 ∼ N (0, 𝐼 ). 
experiments 5.1 experimental settings: 
 Dataset. We utilized a subset of data collected from GRIT [23] and midjourney-v5-202304-clean [30] . Simple filtering was applied to the image resolution and texts, with the total amount of data used for training approximating 1 million. To ensure a fair comparison, our model was trained alongside existing text-encoder-based models (including SD1.5 [27] , SDXL [24] , and T5-based SD1.5 models) using the same dataset and similar Adapters. Base models. Our experiments are conducted based on pre-trained Stable Diffusion (SD) 1.5 model [27] , utilizing two Large Language Models (LLMs), Phi1.5 [17] and Vicuna1.5-7B [6] . The number of parameters of Phi1. Metrics. We assess the models from three dimensions. (1) For controllability, we evaluate the degree of matching between the generated images and the given text via the CLIP Score. However, since the SD model itself is based on CLIP, for fairness, we employ the SigLIP-L-384 [32] model to calculate the SigLIP Score:EQUATION)where 𝐼 is the input image, 𝐿 is the input text, 𝑓 𝑖𝑚𝑔 is the image encoder, and 𝑓 𝑡𝑒𝑥𝑡 is the text encoder. 𝛼 and 𝛽 are the learned parameters from SigLIP model. ( 2) For image quality, we utilize CLIP-IQA [29] to evaluate the quality of the images from the aspects of image details and overall image quality. (3) As for the logicality of images, we employ the user study. For each model, we construct 15 prompts from multiple perspectives, including action logic, color matching, and the number of objects, etc. Each prompt generates 10 images, and human evaluators judge whether the core logic of these prompts is reflected in the images.For a quantitative analysis of our method, we use the SigLIP Score to evaluate how well the generated images match the given text. Furthermore, we use CLIP-IQA to analyze the image's Quality, one white rabbit is standing on a wooden bench near the garden, with two blue cat next to it and one cat is looking at that rabbit.A beautiful butterfly with iridescent wings, its upper wings is shades of blue and purple, while the lower wings is shades of green and yellow, with sparkling diamonds on its body.In a city turned into a wasteland after the Great War, a robot stands on the road next to a rat-headed robot with a rat at its feet.One blue bird with pink mouth is standing on the side of a road, and one pink cat is looking by that bird.In the room, there is a transparent glass vase filled with beautiful pink roses. Three lemons are placed next to the vase on a table. The table is a next to a green couch. The couch is adorned with two pillows, One is pink and the other has a green and gray pattern. The floor is covered with wooden flooring. The walls are painted with latex paint. The ceiling is made of plaster and has a line-shaped design. The room has a large window, providing ample natural light.Figure 4 : In comparison with existing approaches, LLMDiff exhibits superior capabilities in both language comprehension and action understanding. Furthermore, it is proficient in generating images with high-quality details.Complexity, and Beauty, thereby assessing whether the overall quality of the generated image is better. In Tab. 1, our proposed method based on Vicuna-7B achieved a SigLIP Score of 8.5, which is 31% higher than the best existing SDXL model of 6.2. Meanwhile, the model based on phi-1.5 has a 26% improvement compared to the SD1.5 used as our baseline, approaching the level of SDXL. These results suggest that our LLMDiff Adapter can effectively combine the existing LLM and Diffusion models. Allowing the powerful text comprehension capabilities of the LLM to be utilized in the text-to-image diffusion model, thereby generating images with sufficient controllability. The more powerful the LLM, the stronger the controllability it brings.Regarding the quality of the generated images in Tab. 1, our method surpasses existing methods in multiple aspects such as A picture of the city as seen by a cat falling from a great height.A glass ball falls from a height and hits the ground very hard.A pig that can fly. overall image quality, complexity of details, and aesthetic appeal of the image. The Quality score reached 78.6, improving by 2.7% compared to the currently best SDXL model. In terms of the complexity of the generated image features, the Complexity score reached 24.7, with an improvement of 3.3% compared to SDXL. The aesthetic appeal score of the image also reached 92.9, surpassing the existing SDXL by 2.5%. By controlling the generation process of the Diffusion model through the LLM model, we can not only improve the alignment between the image and the text, but also enhance the quality, detail, and aesthetic appeal of the generated images.Our model is evaluated qualitatively from several aspects, encompassing its capacity to comprehend actions, entity relationships, spatial structures, and complex descriptions. As depicted in Fig. 4 , our approach, powered by the robust semantic comprehension capabilities of LLMs, yields a more precise and controllable outcome in the portrayal of multiple entities and their inter-relationships. For instance, in the first column of Fig. 4 , our method accurately generates an image of a white rabbit seated on a wooden bench with two blue cats next to it, demonstrating a refined understanding of entity quantity and color correspondence. Furthermore, it exhibits a precise comprehension of the action wherein only one cat is looking at that white rabbit. In contrast, existing methods struggle to understand inter-entity relationships, like actions.The third column further underscores our method's ability to accurately generate distinct entities as specified in the description, without any feature confusion. Instead, existing models tend to prioritize keyword comprehension and struggle to understand the holistic context of the sentence. We described three entities: a robot, a robot with a rat head, and a rat. A keyword-based approach risks overlooking some entities due to keyword overlap. Existing models leveraging text encoders like those of CLIP or T5 fail to accurately understand these inter-entity relationships and fail to generate images precisely when entity descriptions have keyword overlaps.When it comes to generating complex scenes from extensive text descriptions, prevailing methods struggle to accurately comprehend the provided long text, resulting in generated scenes that often diverge from the description, omitting numerous features. Our method, instead, integrates the power of LLMs into the diffusion model, exploiting LLMs' powerful long-text comprehension capabilities to precisely delineate each part of the scene, as well as the inter-entity relationships. For instance, as indicated in the last column of Fig. 4 , our method can accurately render the complex indoor scene described in the extensive text, encompassing the pink rose, three lemons, a sofa with pillows, and the interior decoration. In contrast, existing methods fall short in accurately generating these complex features, typically producing simpler room scenes with a dearth of detailed features.Furthermore, our method exhibits promising abilities in generating various meaningful detailed features. Our approach is based on the SD1.5 model, but when combined with the remarkable comprehension capabilities of LLM, our LLMDiff Adapter can significantly enhance the overall texture and detail quality of the generated images. The features generated by SD1.5 are typically fragmented, encompassing indistinguishable local features that are challenging to understand. In contrast, our method is capable of generating features with better coherence and more meaningful local features, particularly in complex scenes described by long texts.Existing text-to-image generation models tend to produce visually highly similar image details with the given texts. They are struggle to generate image details that texts does not explicitly indicate, but is necessary for commonsense or reasoning. In our experiments, this is also taken into consideration for model evaluation, in Fig. 5 .In the first two columns of Fig. 5 , our model can accurately understand the number of entities in the description, which is a great challenge for current diffusion models based on text encoders. With the help of LLM's understanding of quantifiers and entity relationships, we can enable the diffusion model to accurately generate the number of entities given in the description text.In the third column of Fig. 5 , the model is tasked with generating an image from the perspective of what a cat would see when it falls from a great height. The subject is not the cat itself, but rather the scene as viewed by the cat. This requires the model have enough capacity of logical reasoning. A model that primarily relies on keywords to interpret sentences can easily produce an image of a cat falling from a great height. Conversely, our method, grounded in LLMs, comprehends user intent well and generates the scene as perceived by the cat, not an image of the falling cat, despite the presence of the keyword "cat" in our description.The fourth column illustrates how our model effectively leverages the LLM's capability to infer the physical rules. The task is to generate an image of a glass ball falling from a great height. In accordance with physical rules, the glass ball will inevitably shatter upon impact, making a lot of internal cracks that should propagate upwards from the point of impact, surrounded by fragments produced by the shattering of the glass ball. Existing methods lack this reasoning ability: the glass balls they generate remain intact, contradicting objective physical rules. Our method, instead, is able to draw these inferences accurately, generating the features implied in the description, including a shattered glass ball and a glass ball that begins to fracture from the bottom due to the impact. These are what text-encoder-based diffusion models fail to comprehend. The last column reveals our model's capacity to utilize LLM's inherent imagination ability and understanding of functions. The goal is to create a pig that can fly, focusing on its inherent ability to fly rather than its state or actions. It is expected that the generated image should depict an animal with a pig's primary features but a body structure adapted for flight. Existing models all generate a pig flying in the sky, and particularly, SD models simply draw a pig floating in the sky, without any imagination about the function of flying. Our model, instead, infers the user's intention from the text and conceptualizes the ability to fly based on LLM's knowledge base. It borrows structure characteristics from common flying animals, like birds, and integrates them into the pig. The pig's ears evolve into larger structures resembling wings, and it reduces to two feet and a smaller size, which are typical bird traits. Our model thus imagines a pig with flying capabilities rooted in real-world logic and pig features, rather than forcibly attaching wings.According to the results in Fig. 6 , which shows the weight distribution across various layers in the new and original cross-attentions, a substantial portion of the original knowledge within the model is preserved. The preservation is less in layers with a higher number of parameters, while layers at both ends, which have fewer parameters, retain more. The newly added rectification module primarily operates in the decoder part of the U-Net. 
conclusion and limitations: 
 We viewed generative LLMs with a transformer-based decoder-only structure as a diffusion model, thereby we can sample implicit text encodings for image generation. We have designed the LLMDiff Adapter to incorporate these encodings into a text-to-image diffusion model, enhancing the model's controllability and reasoning abilities pertaining to commonsense, logic, and physics. The generated images are more realistic, with improved detail and quality. Moreover, our method outperforms existing text-encoder-based methods in various quantitative metrics. Limitations. Our method requires the output from each Transformer block of the LLM, and thereby is incompatible with closedsource models like GPT-4 and Claude 2.  
anonymous authors: 
 Text Prompt: Envision a secluded beach at twilight, where the last golden rays of the sun cast a soft glow on the rugged cliffs. The sky is a tapestry of pink and purple, with the early evening stars just beginning to emerge. Waves gently lap against the shore, where a lone, ancient oak tree stands, its twisted roots spilling out onto the white sand. Nearby, a weathered boat rests in the sand, abandoned, its paint peeling and sails tattered. In the background, a lighthouse perches precariously at the cliff's edge, its beacon flickering to life as the light fades. The scene is one of serene isolation, untouched by the rush of modern life, a timeless testament to nature's quiet majesty.Text Prompt: In the room, there is a transparent glass vase filled with beautiful pink roses. Three lemons are placed next to the vase on a table. The table is a next to a green couch. The couch is adorned with two pillows, One is pink and the other has a green and gray pattern. The floor is covered with wooden flooring. The walls are painted with latex paint. The ceiling is made of plaster and has a line-shaped design. The room has a large window, providing ample natural light.Figure 1 : Comparison of our LLMDiff with DALL-E 3 [2] . The pink texts represents the parts that our model has understood but DALL-E 3 has not. Diffusion Models like DALL-E 3 that are based on text encoders are prone to neglecting details when interpreting long complex texts, and they lack a comprehensive understanding of entity relationships. Instead, our model, which employs a decoder-only LLM, can more effectively grasp semantic and logical relationships between entities. As a result, it generates images that more accurately align with the user's intent.Groundbreaking advancements in text-to-image generation have recently been achieved with the emergence of diffusion models. These models exhibit a remarkable ability to generate highly artistic and intricately detailed images based on textual prompts. However, obtaining desired generation outcomes often necessitates repetitive trials of manipulating text prompts just like casting spells on a magic mirror, and the reason behind that is the limited capability of semantic understanding inherent in current image generation models. Specifically, existing diffusion models encode the input text prompt with a pre-trained encoder structure, which is usually trained on a limited amount of image-caption pairs. State-of-the-art large language models (LLMs) based on the decoder-only structure have shown very powerful semantic understanding capability as their architectures are more suitable for training on very largescale unlabeled data. In this work, we propose to enhance textto-image diffusion models by borrowing the strength of semantic understanding from large language models (LLMs), resulting in a simple yet effective adapter to allow the diffusion models to be 
 
 Appendix: A NEW CONTROLLER FOR TEXT-TO-IMAGE GENERATIONIn this section, we elucidate the theoretical analysis for extracting text encodings from decoder-only LLMs. Initially, we reveal that Transformer-based LLMs can be rephrased as diffusion model. Within this view, we pinpoint a specific timestep in the decoder component of an encoder-decoder LLM to deduce the encoder's text encoding distribution from its input and output. This deduction is then extended to decoder-only models, leading to the conclusion that text encodings can be estimated from the outputs generated for sentences and words at each timestep. 


Decision: This paper proposes a new approach for text-to-image generation aiming to improve the semantic understanding capabilities of the model and thus create a better image. It is clearly a paper that fits ACM Multimedia (it has interesting techniques as well as a user study). All reviewers indicate several positive aspects of the method but also indicate that the method could have done better in comparing to existing baselines. All in all the scores come to 2 borderline accepts and one weak accept. Overall I feel that the paper can be accepted.

Reviews: review: paper_summary: paper_summary: In this paper, the authors propose an approach to text-to-image diffusion models by integrating large language models (LLMs) with a decoder-only architecture. The proposed method aims to enhance semantic understanding and generate higher quality images from text prompts. Traditional text-to-image models use an encoder structure which often struggles with understanding complex, detailed prompts and lacks in generating high-fidelity images. The authors introduce an "LLMDiff-Adapter" that allows seamless integration of decoder-only LLMs with diffusion models, improving image generation quality, reliability, and text comprehension.
strength_weakness: strength_weakness: - The authors provide detailed theoretical analysis in Section 3.4.
- It seems that images generated using the proposed method are superior in quality and detail compared to those produced by state-of-the-art models like DALL-E 3 and Stable Diffusion.
limitations: limitations: - The approach heavily relies on the capabilities of large pre-trained language models, which increases the complexity and requires significant computational resources.
- It would be great if the authors could include and analyse some failure cases.
- There is a lack of discussions on how the proposed method compares to other possible integrations of LLMs with diffusion models that might use different architectural or training strategies.
- There are too few details regarding the user study of this paper.
paper_summary: paper_summary: The paper attempts solving the issue of limited semantic understanding in text-to-image generation models. The authors proposed an adapter to incorporate the knowledge from LLMs into the cross attention conditioning blocks of diffusion models - with the goal being to leverage the rich knowledge these large scale trained LLMs have.
strength_weakness: strength_weakness: Paper is well written and insightful. The direction of approach is interesting, and is different from the existing LLM directions where the utility of LLM has been limited to coming up with just more detailed input prompts. The evaluation protocols to evaluate multiple aspects (e.g. entity relationship, actions) is extensively designed.
limitations: limitations: I am overall positive about the novelty, technical depth and quality of the work. 
The only aspect is lack of comparison with baselines - there are many works in the direction of prompt optimization with or without using LLMs. Would like to see how the proposed method compares with them.
paper_summary: paper_summary: This paper proposes an innovative framework for controlling text-to-image generation using Large Language Models (LLM). The framework connects LLMs and Diffusion Models through the LLMdiff Adapter. The paper provides theoretical analysis as well as qualitative and quantitative experiments to validate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and well-structured. 
2. This paper uses LLM as a text controller to guide the Diffusion Model that does not need further training in the UNet and LLM, which is somewhat innovative.
3. The experimental results show that LLM can outperform the baseline, such as SD1.5 and SD-XL. In some cases, this method outperforms DALL-E 3.
limitations: limitations: 1. Figure 2 requires further clarification. The current version does not clearly demonstrate the advantages of decoder-only LLMs over encoder-decoder structures. 
2. Some experimental settings are not clear. Which dataset are the quantitative experiments conducted on? 
3. The experiments are limited. Lack of comparison on current popular decoder-only architectures, e.g., Llama. Additionally, Pixel-Art α, which also uses T5-XL as a text encoder, does not compare to your method. The text-to-image generation results have not been tested on the T2I-CompBench benchmark.