Equilibrated Diffusion: Frequency-aware Textual Embedding for Equilibrated Image Customization

Authors: liyuan  ma
guo-jun  qi
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-02 08:51:47

Abstract: Image customization involves learning the subject from provided concept images and generating it within textual contexts, typically yielding alterations of attributes such as style or background. 
Prevailing methods primarily rely on fine-tuning technique, wherein a unified latent embedding is employed to characterize various concept attributes. 
However, the attribute entanglement renders customized result challenging to mitigate the influence of subject-irrelevant attributes (e.g., style and background).
To overcome these issues, we propose Equilibrated Diffusion, an innovative method that achieves equilibrated image customization by decoupling entangled concept attributes from a frequency-aware perspective, thus harmonizing textual and visual consistency.
Unlike conventional approaches that employ a shared latent embedding and tuning process to learn concept, our Equilibrated Diffusion draws inspiration from the correlation between high- and low-frequency components with image style and content, decomposing concept accordingly in the frequency domain. 
Through independently optimizing concept embeddings in the frequency domain, the denoising model not only enriches its comprehension of style attribute irrelevant to subject identity but also inherently augments its aptitude for accommodating novel stylized descriptions.
Furthermore, by combining different frequency embeddings, our model retains the spatially original customization capability. 
We further design a diffusion process guided by subject masks to alleviate the influence of background attribute, thereby strengthening text alignment.
To ensure subject-related information consistency, Residual Reference Attention (RRA) is incorporated into the denoising model of spatial attention computation, effectively preserving structural details. 
Experimental results demonstrate that Equilibrated Diffusion surpasses other competitors with better subject consistency while closely adhering to text descriptions, thus validating the superiority of our approach.

Main paper: 
 introduction: 
 Text-to-Image (T2I) customization aims to customize a Text-to-Image diffusion model using a handful of provided concept images to generate diverse images aligned with the target prompts. This technology can support a variety of downstream applications, spanning from virtual photography to personalized e-commerce product design. It is noteworthy that the primary challenge in this task lies in striking a balance between maintaining consistency in the subject identity and coherence with the target prompts.Current T2I customization paradigm mainly encompasses two main approaches: model finetuning and encoder-based pretraining. Encoder-based approaches [12, 43, 47] usually leveraged large-scale datasets tailored to specific domains to learn concepts. However, these methods face challenges in generalization across diverse domains and necessitate substantial image datasets, thereby compromising their adaptability. In contrast, finetuning-based methods [1, 11, 20, 24, 32, 42] established associations between concept and latent textual embedding through denoising optimization. However, textual representations are constrained to overfit given concept and lead to entanglement with identity-unrelated style and background attributes, which undermines its capacity to accommodate new textual descriptions during customization. Although some works [3, 6] aim to mitigate the interference of identity-related information by leveraging distinct embeddings. They primarily focus on spatial attributes such as pose and are unable to address the influence of style attribute. Furthermore, textual representations lack spatial expressiveness, resulting in poor image alignment. We attribute the challenge of achieving the trade-off between imageand stylized prompt-alignment in customization to the interplay between content and style during the optimization. This interplay necessitates the fitting of model parameters representing specific concepts to both content and style, making it difficult to prevent identity-related attribute from being influenced by the style attribute of original concept image.In this paper, we propose Equilibrated Diffusion to address aforementioned issues, which succeed in image-and text-aligned customization as shown in fig. 1 . It incorporates a comprehensive training strategy comprising Frequency-aware Decoupled Textual Embedding (FDTE) and a Mask Guided Diffusion Process (MGDP) to enhance text-alignment by disentanglement between identityirrelevant attributes (e.g., style and background) with identityrelevant one, along with the Residual Reference Attention (RRA) to enhance image alignment. Motivated by the fact that image style and content can be represented by high-and low-frequency components [15] , FDTE decouples the concept into different textual embeddings, each bound to the denoising process of different frequency bands of the input image. Through the independent optimization of content and style embeddings and denoising learning applied to image inputs across various frequency bands, the denoising model not only enhances its understanding of style attributes unrelated to subject identity but also intrinsically augments its ability to accommodate novel stylized descriptions as evidenced by results presented in fig. 8 . Furthermore, MGDP utilizes the mask of subject to restrict concept denoising learning in the subject region, allowing the model to focus on learning the subject's concept and eliminate background interference. Finally, to improve the spatial details preservation of subject concept, our proposed RRA employs a spatial attention mechanism to explicitly inject spatial structural details from encoded reference into denoising features.In summary, our approach aims to achieve subject consistency and alignment with textual descriptions in the customization of image concepts, offering the following contributions. Firstly, we propose FDTE and MGDP in our training strategy to mitigate interference from irrelevant attributes such as style and background on concepts. These techniques aid the model in learning identityrelated concepts while excluding interference from style and background attributes irrelevant to identity. Secondly, we design RRA to preserve texture details of the subject concept in generated images with spatial attention mechanisms. Finally, we demonstrate that Equilibrated Diffusion outperforms prior methods in achieving consistency between identity and prompt, showing qualitative and quantitative advantages. 
related work 2.1 diffusion-based text-to-image generation: 
 Diffusion models [16, 31, 36, 38] constitute a category of generative models that learn to simulate the generation process through sequential denoising process, which restores the noisy data disturbed by forward diffusion. Significant advancements have been achieved in image generation based on diffusion models. Subsequent endeavors have focused on enhancing the controllability of generated outputs by incorporating additional conditional information [41, 44] (such as semantic map, sketch, and text, etc.) to govern the generation process. Among these, text conditioning inherently offers superior editability and flexibility. Concurrently, with advancements in large language models, the understanding of text has also improved, thereby fostering the development of text-to-image generation models [25, 29, 33] . Nevertheless, despite the potential for diverse image generation guided by text, textconditioned image generation models encounter difficulties in tailored image synthesis related to specific concepts. Customizing images entails preserving the identity information correlated with the provided textual descriptions, posing a significant challenge for text-to-image generation models.Leveraging the power of text-to-image diffusion model, image customization [1-3, 6, 10-13, 20, 24, 32, 42] has witnessed rapid development in controllability and faithfulness. Given a set of concept images, image customization is designed to produce new images of a given subject that adhere to textual descriptions. The core of image customization largely revolves around the adjustment of parameters within the foundational text-to-image generation model or incorporating additional parameters. Such modifications involves fine-tuning text embedding [1, 11, 42] , adjusting parameters within the full or partial U-Net network [8, 23, 32, 39] , as well as integrating extra encoder components [7, 9, 14, 18, 34, 43, 45, 47] . Early methods [1, 10, 11, 13, 42] capture the concept of the subject through text embeddings represented by specific token. Typically, Textual Inversion [11] encapsulates the concept with optimized token embedding and generates the customized concept image by composing the optimized token with target prompt. Dream-Booth [32] and Custom diffusion [20] has opted for fine-tuning the U-Net parameters, which results in improved subject fidelity. Another line of work has focused on exploring fast customization generation by additional encoder which requires substantial data and usually focuses on close-domain datasets such as human face. Our work mainly involves general customization and thus is compared with open-domain method Elite [45] in the experiment. In response to the limited expressiveness of spatial structures in text embeddings, several methods [4, 17, 24, 37] have explored injecting concept information at the self-attention layer. However, the generated results tend to entirely retain the reference subject, leading to the loss of text alignment. Recently, some works [3, 6] propose to distinguish identity-irrelevant attributes from identity-relevant ones. While they mainly focus on the impact of pose and background on the subject, our method prioritizes stylistic attributes. 
preliminary: 
 Diffusion Models. Our method is based on the pretrained Stable Diffusion (SD) [31] , which is a popular text-to-image model and comprises two main components. Initially, an autoencoder with encoder E (•) and decoder D (•) learns to compress input image 𝐼 into a lower-dimensional latent space 𝑧 0 = E (𝐼 ), which is then decoded back into image D (E (𝐼 )) ≈ 𝐼 . Subsequently, a conditional diffusion model 𝜖 𝜃 is trained on this latent space to denoise noisy latent codes conditioned on textual input 𝑦. The training process involves employing a basic mean-squared loss, denoted as:EQUATIONwhere 𝜖 ∼ N (0, 𝐼 ) signifies unscaled noise, 𝑧 𝑡 denotes the latent state at time step 𝑡. During inference, Gaussian noise 𝑧 𝑇 is progressively diminished to 𝑧 0 and then decoded back into image output. Conditional diffusion model 𝜖 𝜃 of SD is based on a U-Net architecture, which comprises convolution layers, cross attention layers and self attention layers [40] . Among them, the self attention layers capture the spatial relationship of image features within themselves. Formally, after projecting the latent features in timestep 𝑡 into queries 𝑞 𝑡 , keys 𝑘 𝑡 and values 𝑣 𝑡 , the self attention calculation is expressed as follows:EQUATION)where 𝑑 is the feature dimension of 𝑞 𝑡 , 𝑘 𝑡 , and 𝑣 𝑡 .Image Customization. Image customization [1, 11, 12, 20, 24, 32, 43] learns a trainable text embedding 𝑆 * to represent the specific concept based on user-provided concept images. Leveraging 𝑆 * , novel images embodying the desired concept can be synthesized under diverse textual descriptions. Previous methods [20, 24, 32] further enhance concept representation during optimization by updating weights of projection matrix of key and value in the crossattention layer or the entire U-Net's parameters. To effectively integrate 𝑆 * into the generative process, the input text prompt 𝑦 is formulated as 'Photo of a 𝑆 * [class]', where [class] corresponds to the specific class name of the designated concept.Image Fourier Transformation. The Fourier transform of images constitutes a fundamental technique widely used in image processing and computer vision fields [19, 22, 27, 35] . It serves as a powerful tool for analyzing the frequency components of an image and extracting important features for various applications. For an image 𝐼 ∈ 𝑅 𝐻 ×𝑊 ×𝐶 , the Fourier transform is defined as:EQUATIONwhere F denotes the frequency-domain representation of the image and F -1 signifies the corresponding inverse Fourier transformation.High-pass and low-pass filters are conventionally applied to the frequency-domain representation of images to isolate specific frequency bands. The high-pass filter, denoted as 𝐻 (𝑢, 𝑣), serves to suppress low-frequency components while retaining high-frequency details. Conversely, the low-pass filter 𝐿(𝑢, 𝑣) is designed to preserve low-frequency information while mitigating high-frequency noise. These filters are formulated as follows:EQUATIONwhere (𝑢 0 , 𝑣 0 ) represents the center of the Fourier frequency map and 𝑓 𝑐 is the cutoff frequency. In our work, we utilize Fourier transformation to acquire conceptual images characterized by high and low frequencies, subsequently leveraging them to enhance the model's denoising capacity across various frequency spectra. 𝑜𝑟𝑖 corresponding to the original image serves as a conditional input to ensure that the generated outcomes adhere closely to real images. 
mask guided diffusion process: 
 While FDTE aids in diminishing the model's reliance on the original image style, direct learning of conceptual representations during the image diffusion process may be interfered by subject-irrelevant background attribute. Although previous works [2, 20] have explored strategies wherein the computation of diffusion loss is confined exclusively within the subject region guided by accurate mask, the generated results are not promising as shown in fig. 8 . Inspired by SmartBrush [46] , we propose a mask-guided diffusion process guided by the subject mask 𝑀 𝑔 , thus compelling the model to focus on learning the conceptual representation of the subject itself. Specifically, as shown in fig. 3 , we conduct noise addition and prediction within the subject mask region in the diffusion process and retain clean background information inputted into UNet, which prompts the learning of subject concept. Mathematically, the input The aggregated reference attention map closely reassembles the ground truth subject mask during L 𝑟𝑎 calculation eq. ( 10). (c) Reference Mask 𝑀 𝑓 predicted in RRA accurately distinguishes the subject from the background. noisy latent of UNet is formulated as follows:EQUATIONFinally, we reformulate the denoising objective to be aligned with masked diffusion loss [2] and express it as follows:EQUATION. (7)As illustrated in fig. 4 , we propose Residual Reference Attention to enhance image alignment by incorporating spatial details of the subject from reference image with reference attention. In the training process of the model, the features of the reference image 𝐼 𝑟 are initially encoded to obtain the latent code 𝑧 𝑟 0 , which is then perturbed with Gaussian noise of 𝑡 step to obtain 𝑧 𝑟 𝑡 . This noisy latent code is then fed into a frozen SD UNet 𝜖 𝑟 . Before the selfattention module of 𝜖 𝑟 , reference features are extracted and merged with the corresponding features at the positions of UNet 𝜖 𝜃 containing trainable parameters through RRA, ensuring the proximity and effectiveness of feature fusion in the feature space. During the inference phase, features from the reference image are injected into the denoising network, gradually incorporating spatial information from the reference image during the denoising process over 𝑇 steps.In typical personalized tasks, occlusions and variations in viewpoint often occur between reference images and generated results. Consequently, simple fusion of reference image features may introduce interference. Previous approaches [17, 24] only focused on the foreground regions of the subject, neglecting the influence of backgrounds and mismatched foregrounds, or requiring additional forward processes [4] during inference to obtain the subject's mask for fusion. To address this issue and mitigate additional inference overhead, RRA learns explicit fusion masks and adaptively determines fusion coefficients during the optimization process. As illustrated in Figure fig. 4 , RRA projects reference appearance features 𝐹 𝑟 ∈ R ℎ𝑤×𝑑 into keys and values, sharing the same queries from target features 𝐹 𝑔 ∈ R ℎ𝑤×𝑑 within the corresponding selfattention modules. Subsequently, the attention computation results from both reference attention and self attention modules are concatenated as inputs for reference mask 𝑀 𝑓 prediction and feature calculation, which is formulated as followsEQUATION)where ⟨•⟩ and [•] mean the feature splitting and concatenation operations, respectively. 𝜙 (•) denotes the clamp function, constraining the results to the interval (0, 1) for generating reference mask 𝑀 𝑓 as visualized in fig. 5 (c ). Reference attention mask 𝑀 𝑓 ∈ R ℎ𝑤×1 and reference feature 𝐹 𝑟 𝑅𝐴 ∈ R ℎ𝑤×𝑑 are calculated by linear layer H 𝑟 : R 2𝑑 → R 𝑑+1 , which is trainable and zero initialized to promote the training stability at the beginning. The target feature 𝐹 𝑔 𝑆𝐴 is acquired by frozen linear layer H 𝑠 : R 𝑑 → R 𝑑 . The reference attention output can be regarded as the residual information for appearance reference, which is adaptively fused to make the best use of accessible information and is conducted as follows:EQUATION)Reference Attention Loss. The preservation of subject details is built on the precise correspondence learned by reference attention. To facilitate the target feature attend more relevant area in reference feature map, we introduce the Reference Attention Loss that encourages the target features within the subject area to have a high attention score distributed in the corresponding subject area of paired reference. The detailed formulation of the loss function is denoted as follows:EQUATION)where reference attention map A ∈ R ℎ𝑤×ℎ𝑤 is calculated by the target feature 𝐹 𝑔 and reference feature 𝐹 𝑟 . 𝑀 𝑟 ∈ R ℎ𝑤×1 and 𝑀 𝑔 denote the subject foreground mask of reference and target image extracted by Grounded-SAM [30] , respectively. ⊙ represents the Hadamard product. As depicted in Notably, the visual result of them closely aligns, thereby underscoring the effectiveness of reference attention loss. This affirms its capability to facilitate reference attention in better leveraging the subject information present in the reference image, consequently fostering identity preservation. 
experiments 5.1 experimental settings: 
 Datasets. Our experiment is conducted at DreamBooth dataset [32] , which comprises 30 subjects spanning various categories, including both non-live and live subjects. In addition, we also include an commonly used case called "cat toy," which has been utilized in many previous works [8, 11, 24, 45] . Each subject consists of 4 to 6 images paired with 55 unstylized prompts from [24] and another 22 stylized prompts generated by LLM [26] for customization. A complete prompt list is provided in supplementary material.Metrics. Our evaluation follows the same setup and calculation procedures as CustomDiffusion [20] , which employs three common metrics to assess the effectiveness of models in aligning text and images: CLIP-T, CLIP-I, and DINO-I. CLIP-T measures textalignment by the feature similarity between the CLIP [28] visual representation of the generated image and the CLIP textual representation of the corresponding prompt. For the image-alignment assessment, CLIP-I and DINO-I are used to evaluate the visual similarity between generated image and concept image using the features extracted from pretrained CLIP and DINO [5] models.Implementation Details. We follow previous method [32] to employ Stable Diffusion v1.4 as the text-to-image model for fair comparison. Our model is trained by an A100 GPU with the AdamW optimizer [21] for 600 steps and the learning rate is set to 1e-5. Data augmentation techniques are applied to reference images to enhance the disparity with the denoising image, including image cropping, rotation, and flipping, etc. During the inference phase, the images are generated using 50 steps of DDIM [36] with classifierfree guidance scale set to 12.5.The qualitative comparison results with prior methods are presented in fig. 6 . As can be seen, our method achieves superior customization performance in stylized prompt alignment and subject identity preservation. For textual stylization alignment, our approach excels in handling various challenging stylistic descriptions compared to existing methods. In cases where the style description closely resembles the original image, our method adeptly preserves the local details of the original image while undergoing stylization (e.g, the can in the 1st row of fig. 6 ), thereby enhancing its identity consistency throughout the stylization process. In contrast, for more unrealistic stylistic descriptions, our method adeptly maintains crucial semantic elements of the subject while enabling flexible and intricate style editing, a tradeoff that remains elusive to other methodologies. For instance, unlike other approaches, our method successfully disentangles from the influence of the original image's style (see the 3rd, 5th, and 7th rows of fig. 6 ), thereby avoiding the tendency to generate outcomes resembling the original colors and style, thus demonstrating adaptability to novel stylistic descriptions. Regarding the preservation of subject identity, our method leverages the proposed RRA technique, effectively retaining the crucial texture structures of the concept. More qualitative results can be found in supplementary material. Following this, we conduct a comprehensive evaluation of our method from a quantitative perspective to validate its efficacy. The baseline method for our approach is the custom diffusion. As depicted in table 1, we achieved the highest CLIP-I and DINO-I scores with both stylized and unstylized textual descriptions. This indicates that our method excels in preserving texture information of the subject and maintains superiority in retaining subject identity. Furthermore, our method demonstrates close-to-leading results in stylized scenarios compared to Dreambooth, while exhibiting optimal CLIP-T score in non-stylized prompts, showcasing its ability for text alignment. This showcases our method's capability for superior textual editability. In summary, our method's leading performance across these metrics attests to its effectiveness in preserving subject identity and aligning with textual descriptions.In this section, we conduct ablation study on the core components of our method to verify their contributions. Visual comparison results of various ablation models are presented in fig. 8 , while numerical analyses are provided in fig. 7 . The evaluation settings of each experiment remain the same as in section 5.1.Effect of Residual Reference Attention and L 𝑟𝑎 . Compared to the baseline, the visual results in fig. 8 indicate that the variant model B better preserves the main structural details in conceptual images after incorporating RRA, such as the patterns on jars or the appearance of cats. Additionally, improvements are observed in metrics like CLIP-I and DINO-I, which indicate enhanced subject consistency, highlighting the significant efficacy of the proposed RRA. Analysis of the reference attention heatmaps further reveals that the generated results accurately attend to corresponding positions in the reference images, validating the effectiveness of the RRA.Moreover, reference attention loss L 𝑟𝑎 further improves imagealignment metrics. Visual analysis of the aggregated reference attention maps demonstrates that the subject area can effectively focus on the corresponding areas in the reference, as evidenced by the comparison between the aggregated reference attention map To investigate FDTE's hyperparameter settings in detail, we analyze the probabilities of selecting high-frequency, low-frequency, and original images. For unstylized and stylized scenarios, using 𝑝 𝑙 , 𝑝 ℎ , 𝑝 𝑜 = [0.1, 0.1, 0.8] achieves superior metrics, indicating an optimal balance between image fidelity and textual coherence.Effect of Mask Guided Diffusion Process. Finally, we analyze the efficacy of MGDP in mitigating the interference of background on the generated results. As illustrated in fig. 8 , RRA tends to capture spatial information from concept images directly, thereby rendering the background more similar to the conceptual image (see A and B in fig. 8 ). Exclusion of MGDP leads to overfitting to the concept Notably, our goal is to strike a balance between aligning images and text. Although our full setting shown in the upper table of fig. 7 does not yield optimal scores across all metrics compared to other model variations, the lower figure of fig. 7 reveals that our approach is positioned closer to the upper-right quadrant. This observation demonstrates the superiority of our method in achieving the desired trade-off between image-and text-alignment. 
conclusion: 
 In conclusion, we propose Equilibrated Diffusion to customize images for better image consistency and stylized text alignment. Specifically, by decoupling content and style through frequencyaware decoupled textual embedding, we decompose the original diffusion optimization process across different frequency bands. This enhances the model's ability to understand content represented by low frequencies and style represented by high frequencies, which is guided by decoupled text embeddings and enhances text consistency expression. The mask guided diffusion process mitigates the influence of the concept image background on the results and further enhances text alignment. Moreover, the residual reference attention and reference attention loss better transfer spatial details from reference concepts, promoting texture consistency. 
 
 Appendix:  


Decision: The submission addresses the challenging problem of disentangling content and style in image customization tasks. The proposed method introduces innovative solutions such as frequency-aware decoupled textual embeddings and residual reference attention to tackle these challenges effectively. All reviewers have raised valid points, and the authors have satisfactorily addressed most concerns. The method's novelty and the strong experimental results make a compelling case for acceptance. Thus, I recommend a Accept (Poster) for this submission, acknowledging its contributions to multimedia/multimodal processing while suggesting improvements in clarity and comprehensive evaluation in future revisions.

Reviews: review: paper_summary: paper_summary: The paper focus on the issue of the entanglement between the content and style in image customization task. They learn two embeddings in the high and low frequency band respectively to disentangle the content and style. They also propose residual reference attention to enhance image alignment by incorporating spatial details of the subject from reference image.
strength_weakness: strength_weakness: 1.The method of learning two embeddings in the low and frequency band respectively is novel and suitable for the entanglement issue.
2.The performance of the method is good in the experiments.
limitations: limitations: There are something that doesn’t make sense in the Residual Reference Attention.
1. Why the attention computation results from both reference attention and self attention modules should be concatenated as inputs for reference mask prediction. It may be enough to predict the reference mask using the reference attention only and there is no ablation study to prove the effectiveness of the concatenation.
2. What’s more, it’s strange to to sum the production of the reference mask and the reference image and the production of the inverse mask and the target image.
paper_summary: paper_summary: The presented paper tackles the image customization problem. The aim of this task is to learn a subject from some concept images and generate it within a textual context. Previous methods struggle to disentangle subject-irrelevant attributes from the concept itself. In this setting, the proposed approach leverages the correlation between high- and low-frequency components with image style and content to disentangle the concept information. The presented methods propose a diffusion process guided by subject masks to alleviate the background influence.
strength_weakness: strength_weakness: - ablation experiments, quantitative, and qualitative results support the paper's claims
- the paper presents a user study that confirms the claims
- frequency analysis already proved to be a helpful instrument in the image generation [1] task, therefore supporting the claim


[1] Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., ... & Lempitsky, V. (2022). Resolution-robust large mask inpainting with Fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 2149-2159).
limitations: limitations: - [547] It is not clear what M^r is. There are no references in the figures or in the text. Is it perhaps M^f
- Figure 7 uses abbreviations for the names of the methods (i.e., A, B, C, D), but these abbreviations are not defined in the text. I suppose they are defined in Fig. 8 (a). Nonetheless, I advise you to define them before using them.
- More qualitative examples in the supplementary can help the reader to understand the goodness/limitations of the proposed method
paper_summary: paper_summary: The paper presents Equilibrated Diffusion, a novel architecture tackling the task of image customization. The authors state that the main challenge regards the entanglement between content and style, and previous methods fail to decouple these two aspects. To solve such issues, the authors introduce various components in their architecture. In particular, a frequency-aware decoupled textual embedding separates high-frequency style from low-frequency content, thus reducing the reliance on the original image style and allowing the representation of diverse styles. In addition, mask guidance helps to avoid interference from the background, while Residual Reference Attention (along with a reference attention loss) is applied to incorporate spatial details of the reference image to preserve the subject consistency. 
The proposed method is evaluated both qualitatively and quantitatively against state-of-the-art approaches, showing promising results in terms of both subject preservation and alignment with different textual prompts representing diverse styles and contexts. An ablation study assesses the contribution of each architectural component.
strength_weakness: strength_weakness: ## Strengths

* The paper is well-presented and structured. I appreciated the good analysis of the current challenges and the problems faced by previous approaches. 

* The idea of using frequency-aware Decoupled Textual Embedding and residual reference attention is interesting and well-motivated.

* Overall, the experimental evaluation is well presented, and the model is evaluated considering different aspects: qualitative and quantitative comparison against several SOTA methods, image-text alignment, ablation study to investigate the contribution of each single component. Results are also validated through a user study that corroborates the superiority of the "Equilibrated Diffusion" model.
limitations: limitations: ## Major Weaknesses

* I believe that including a section on limitations would benefit the quality of the paper. In this paper, such a section is missing. In particular, I am interested in the possible failure cases of the model. Do the authors have any idea for which cases RRA and  FDTE could fail? I would appreciate a discussion that goes beyond the fact that the model could inherit the classical limitations of Stable Diffusion (e.g. hands, text,...), but instead mainly focuses on the mentioned components of the proposed architecture.  

* Although the experimental evaluation is extensive and well conducted, metrics to evaluate the final image quality are missing (e.g. FID [1], KID [2],...)

[1]: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, NeurIPS 2017.

[2]: Demystifyng MMD GANs, ICLR 2018

## Minor Weaknesses

Additional comments to improve the overall quality of the paper (not influencing the final recommendation):

* In Equation 8 (last row), should not the term be $F^g_{SA}$  instead of $F^{r}_{SA}$? 

* Typo line 579: ”assessment” instead of ”assessment”.