Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis
Authors: Zhiyu Jin
Xuli Shen
Bin Li
Xiangyang Xue
Venue: NeurIPS 2023 poster (2023)
Keywords: Text-to-Image Synthesis
Variable-Sized Image Synthesis
Entropy

Abstract: Diffusion models (DMs) have recently gained attention with state-of-the-art performance in text-to-image synthesis. Abiding by the tradition in deep learning, DMs are trained and evaluated on the images with fixed sizes. However, users are demanding for various images with specific sizes and various aspect ratio. This paper focuses on adapting text-to-image diffusion models to handle such variety while maintaining visual fidelity. First we observe that, during the synthesis, lower resolution images suffer from incomplete object portrayal, while higher resolution images exhibit repetitively disordered presentation. Next, we establish a statistical relationship indicating that attention entropy changes with token quantity, suggesting that models aggregate spatial information in proportion to image resolution. The subsequent interpretation on our observations is that objects are incompletely depicted due to limited spatial information for low resolutions, while repetitively disorganized presentation arises from redundant spatial information for high resolutions. From this perspective, we propose a scaling factor to alleviate the change of attention entropy and mitigate the defective pattern observed. Extensive experimental results validate the efficacy of the proposed scaling factor, enabling models to achieve better visual effects, image quality, and text alignment. Notably, these improvements are achieved without additional training or fine-tuning techniques.

Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis
Zhiyu Jin', Xuli Shen 1.2, Bin Li Xiangyang Xue
1 Shanghai Key Laboratory of Intelligent Information Processing School of Computer Science, Fudan University UniDT Technology
Abstract
Diffusion models (DMs) have recently gained attention with state-of-the-arr per- formance in text-to-image synthesis. Abiding by the tradition in deep learning, DMs are trained and evaluated on the images with fixed sizes. However, users are demanding for various images with specific sizes and various aspect ratio. This paper focuses adapting text-to-image diffusion models to handle such variety while maintaining visual fidelity. First we observe that, during the synthesis, lower resolution images suffer from incomplete object portrayal, while higher resolution images exhibit repetitively disordered presentation. Next, we establish a statistical relationship indicating that attention entropy changes with token quantity, suggest- ing that models aggregate spatial information in proportion to image resolution. observations is that objects incompletely depicted due to limited spatial information for low resolutions, while repetitively disorganized presentation arises from redundant spatial information for high reso- lutions. From this perspective, we propose a scaling factor to alleviate the change of attention entropy and mitigate the defective pattern observed. Extensive experi- mental results validate the efficacy of the proposed scaling factor, enabling models achieve better visual effects, image quality, and text alignment. Notably, these improvements are achieved without additional training or fine-tuning techniques.
The subsequent interpretation
Introduction
Diffusion models have emerged as powerful technique image synthesis [0. 15 46 achieving
state-of-the-arr performance in various applications [12 28 14]
Among them, text-to-image image diffusion
models have garnered significant attention and witnessed a surge in demand [40 37 36). Traditionally, diffusion models have adhered to the typical deep learning approach of training and testing images with predetermined sizes, which generally leads to high-quality results. They still exhibit a range of visual defects and diverse flaws when confronted with novel synthesizing resolution (e.g., 5122 in training while 224 in testing). However, real-world scenarios often demand the generation images with diverse sizes and aspect ratios, necessitating models that can handle such variety with minimum loss in visual fidelity. The necessity becomes even more severe in the generation of large models. As the size of models continues increase, the associated training costs experience substantial surge, thereby posing financial challenges for average programmers and emerging startups, making it unfeasible for them to train specialized models tailored their specific needs. Consequently, there is an urgent demand to explore methodologies that facilitate the full utilization of open-sourced models trained on fixed sizes.
In regard to this limitation, our first key observation reveals that most instances poor performance could be attributed to two prevalent patterns: incomplete or inadequately represented objects and
"Coresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A peraws on the Mows
roj Sofail Delane
Fornst svautiais road
(d) Age woustute
(e) M2X Glltter Chean calen GCP22 Edes Brand PACK
BMW X3 Shop Setf
Colorfal house aviitrr nanere hnclgrownd
Figure 1: Synthesized results with the proposed scaling factor (marked with yellow icons) and those with original scaling factor (marked with green icons). The method with our scaling factor successfully synthesizes high-fidelity and natural images for different resolutions. Please zoom in for better visual effect.
repetitively disordered presentations. We have included several examples of flawed synthesized images in Figure I designated by green icons. Note that smaller images (e.g.. images (a), (b), (c) and (d)) conform the first pattern, showcasing inadequately depicted objects. Conversely, larger images (e.g., images (c), (f) and (g)) exhibit the second pattern, generating disordered objects in a repetitive manner. This delineation, also observed in other problematic cases, allows formulate second key observation, where lower resolution images are more vulnerable to the first pattern, while susceptibility to the second pattern increases in higher resolution images.
In this work, we tackle the challenge of adapting text-to-image diffusion models to proficiently synthesize images spanning wide range of sizes, encompassing both low and high resolutions. Our goal is threefold: (1) achieve a higher level of fidelity regardless of synthesized image resolutions; (2) mitigate the abovementioned two patterns; and (3) augment the semantic alignment between the synthesized images and the text prompts.
To accomplish these, our approach centers around the concept of entropy, which measures the spatial granularity of information aggregation. Specifically, when the attention entropy rises, each token is attending to wider spatial information, otherwise the opposite. Bearing that in mind, we establish the statistical relationship between the resolution (the number of tokens) and the entropy of attention maps. Our finding signifies that the attention entropy varies in correspondence with the quantity tokens, which implies that models are aggregating spatial information in proportion to resolutions. We then establish our key interpretations based on the proportionality. In particular, since narrower attention is paid during the synthesis low resolution images, models encounter difficulties in representing complete objects with limited spatial information. Conversely, when synthesizing high resolution images, each token tends to conduct wide spatial aggregation which results redundancy of analogous spatial information and the disordered presentations of repeated elements.
Based on the statistical relationship and interpretations, we propose a novel scaling factor specifically designed for mitigating the change of the entropy in visual attention layers. We conduct qualitative and quantitative experiments to validate the efficacy our scaling factor, which synthesizes results of varied resolutions with better visual effect and achieves better scores in image quality and text alignment. It is worth noting that the improvement implemented in a training-fres manner by replacing a single scaling factor in the attention layers.
The contributions are summarized follows:
We observe two distinct patterns of defects that low resolution images prone impaired object portrayal while repetitively disordered presentation exhibits high resolution images.
We establish a statistical relationship between attention the entropy and the resolution, which interprets the patterns by ascribing them to the lack or the surplus of the spatial information.

---

We propose scaling factor to efficiently improve variable-sized text-to- cext-to-iman image synthesis in training-free free manner. We conduct various experiments and analysis validate the efficacy.
2
Related Work
Image super-resolution and extrapolation. An alternative path to generate new images with different resolutions is post-processing synthesized images with image super-resolution and extrapo- lation methods. Super-resolution methods [18 53 [5] dedicate to exploiting the high frequency information of images and extending both the height and width of images by fixed factor, which cannot change the aspect ratio of images. For extrapolation methods, they do have the ability to
change the aspect ratio by extending images beyond their boundaries (50] 58 29] However, these
methods are effective for signals that exhibit an intensive stationary component (5) (e.g., landscapes) while we would like to generate images with various scenes. Additionally, extrapolation methods might have discrepancy with diffusion models, resulting in inharmonious visual effect. Instead of
adding more methods for post-proe proteprocessing and making the pipeline lengthy, we seek to synthesize high-fidelity images with diffusion models alone.
Diffusion models and attention mechanism. Diffusion models have emerged as promising family
of deep generative models with outstanding performance in image synthesis [10 [5] 22
45
46
47]
They have significantly revolutionized computer vision research in multiple areas, including image
editing [8 12 inpaintng 28 39 (6), super-resolution [16 4T and video synthesis 14 17].
These models generate high-quality images by smoothly denoising a noisy image sampled from standard Gaussian distribution through multiple iterations. A key component in diffusion models is the attention module. Previous works such as denoising diffusion probabilistic model [15], denoising diffusion implicit model (46), latent diffusion model (37 have demonstrated the capability empowered by the attention modules. Note that attention modules in diffusion models are dealing with thousands of visual tokens and one of the most well-known research concern of attention how to lower its conventional quadratic complexity against the large token numbers. To deal with this, most works define a sparse attention pattern to restrict each query to attend to a limited subset of tokens exclusively. The sparse attention pattern could be either defined as fixed windows 23 52 49 38 30)
or learned during training (25] 34 7 4 660 26 57 [5] 9 20). We draw upon the insights in these
works for sparsity controlling and seek efficiently adapt trained models for variable-sizel data.
Attention entropy. Introduced in the information theory by entropy serves as measure for information and uncertainty. Different research works have introduced entropy to attention maps in various tasks, including semantic segmentation (24). text classification (2), text generation [48), active visual exploration [3] and interpretability analysis [2]] Among them, the work in text generation 48) closely aligns with
attention scaling, leveraging Transformers to accommodate sentences of variable lengths. We draw upon the innovations in the entropy of attention maps for information quantification and seek to exploit both of its practicability and interpretability.
Method
This section establishes the connection between the attention entropy and the token number in a statistical manner and presents scaling factor replaced into diffusion model for variable-sized image synthesis. Our aim is alleviate the fluctuation in attention entropy with varying token numbers.
3.1 Connection: the attention entropy and the token number
Diffusion models refer probabilistic models that learn a data distribution by iteratively refining a normally distributed variable, which is equivalent to learn the reverse process of a fixed Markov Chain with a length of T. In image synthesis, the most efficacious models make use of a Transformer- based encoder-decoder architecture for the denoising step. This step can be understood as an evenly weighted chain of denoising autoencoders, which are trained to predict a filtered version of their input via cross-attention. Yet the scaling factor inside the attention module of diffusion models is so far an under-explored area of research. In this section, we focus on enhancing conditional diffusion models by adaptively determining the scaling factor of Transformer encoder-decoder backbones.
3

---

Let X E I Nxd denotes an image token sequence to an attention module, where N, d are the number of tokens and the token dimension, respectively. We then denote the key, query and value matrices as K XW Q and V. where w E IRddd is a learnable projection matrix and d, the projection dimension. The attention layer (3 computes Attention Q K, V) = AV with the attention map calculated by the row-wise softmax function as follows:
Aij
Σ 2j'=1 A
where À a scaling factor, usually set as 1/v in the widely-used Scaled Dot-Product Attention [5] and i are the row indices on the matrices Q and K, respectively. Following [1] we calculate the attention entropy by treating the attention distribution of each token as a probability mass function of a discrete random variable. The attention entropy with respect to the zth row of A is defined as:
Ent A.,)
Σ A. log(A..)):
where the attention entropy gets its maximum when A., = I/N for all j and gets its minimum when A = 1 and A 0 for any # j'. It implies that the attention entropy measures the spatial granularity of information aggregation. larger attention entropy suggests that wider contextual information is taken into account while smaller entropy indicates only few tokens are weighed.
We now investigate the relation between the attention entropy and the token number. By substituting Eq. ① into Eq. we have:
QQ,K
eA,K
Ent(A:) Σ j=1
log
Σ3=1 1Q.KT.
=1
(3)
E, (AQ,K] e esO.K))
log N + log E, (c.Q.KT)
E, (e,Kk)))
where E, denotes the expectation upon the index j, and the last equality holds asymptotically when N gets larger. For the complete proof, please refer to the Supplementary Materials.
Given X as encoded sequence from image, each token X, could be assumed as vector sampled from a multivariate Gaussian distribution N N(xx Σ²x This prevailing assumption is widely shared
in the fields of image synthesis [13] and style transfer I9 27 21. In this way. each K
X, w could be considered as a sample from the multivariate Gaussian distribution N N( K Σ
N(u> w (W*) " w*). In regard Kj. its dimension-wind linear combination y/ could be treated as a scalar random variable sampled from the univariate Gaussian distribution ~ (H,, = ( (T "T,"Q ). Under this circumstance, we dive into the expectations E, (e)0.K) and E, (AQ,K) 2..K) in B3)
AQ
(Q,K" ) = Ey (c"t) eli+ +f
E, EAQ, e 2Q. = E, (Uiet)) = (Ms + e For the complete proof, please refer the Supplementary Materials. By substituting Eq.44 33) we have:
into
Ent(A.) = log N + log (e)++ (Mi + o
e
(5)
o² 2
= log N
The observation is that the attention entropy varies correspondingly with N in logarithm magnitude while the latter item of = is not relevant with the token number N in the setting of images generation. This implies that each token intends to adaptively deal with wider contextual information for higher resolutions, and narrower contextual information for lower resolutions.

---

Interpretations on two observed patterns. Our key insight lies in the correlation between the fluc- tuating attention entropy and the emergence of two defective patterns in our observations. Specifically, when the model is synthesizing images with lower resolutions, the attention entropy experiences sta- tistical decline of log N, leading reduced atten- tion on contextual information. Consequently, each token is processing discrete information in isolation and the model struggles to generate objects with intricate details and smooth transitions, giving rise to the first observed pattern.
Denoising Step
QK
QK
ATT(Q,KVV) = softmax ((QK
logr N
V A
Replace Scaling Factor
Conversely, in scenarios involving higher resolu- tions, the attention entropy exhibits an upward trend, accompanied by the utilization of over- whelming amounts of global information for syn- thesis. Consequently, each token is processing anal- ogous information and the model generates repet- itively disordered presentations, exemplifying the second pattern.
The illustration of our proposed pro- cess of replacing the scaling factor without ad- ditional training, where T and N denote the number of tokens in the training phase and the inference phase.
Figure 2:
3.2 A scaling factor for mitigating entropy fluctuations
To alleviate the fluctuating entropy with respect to different token numbers N in the inference phase, considering the quadratic form 55). we the scaling hyper-paramesn À in the following form:
x= N,
where e denotes a constant number. Note that when the token number N is set as the number of tokens during training (denoted as 7), À should specialize to 1/vd for the best stablization of entropy (i.e., e/logT 1/vd). Thus, we have an approximate value of c and substitute into 6q.
Vlog N -V---/00 logr N d
A
dlogT
Intuitively, if the model is dealing with token number N >> T, the proposed scaling factor would multiply the feature maps by factor of squared log, N to maintain the rising entropy, and vice versa. Thus, the proposed scaling factor is implemented in training-free manner, which enjoys better performances in synthesizing images of different resolutions. The illustration our proposed method is shown in Figure 2]
4 Experimental Results
In this section, we evaluate our proposed method on text-to-image synthesis and analyse the efficacy of the proposed scaling factor. The quantitative and qualitative results demonstrate better performance of our scaling factor. Please refer to the Supplemental Materials for detailed experimental results.
4.1 Text-to-image synthesis
Experimental setup. We explore the adapting ability of the proposed scaling factor for diffusion models in variable-sized text-to-image synthesis. In particular, we replace the scaling factor of self- attention layers within diffusion models and evaluate their performance without any training upon subset of LAION-40OM and LAION-50 dataset (143 42), which contain over 400 million and 5.85 billion
image-text pairs, respectively. We randomly choose text-image pairs out from each dataset and generate images corresponding to texts in multiple settings for evaluation. As for the diffusion models, we consider both Stable Diffusion B7 trained on LAION-55 5B and Latent Diffusion 37 trained on LAION-4006 to test the performances. The former (which is set to synthesize images with 512 512 resolution by default) maintains 4x longer sequences compared with the latter (with a default 256 x 256 resolution) across all the layers. For more evaluation results and details, please refer to the Supplementary Materials.
5

---

Table 1: FID scores (4) for Stable Diffusion and Latent Diffusion in different resolution settings. Stable Diffusion scores 19.8168 for default 5122 and Latent Diffusion scores 20.4102 for default 256².
Stable Diffusion 512
Latent Diffusion 256
Resolution
Original
Ours
Resolution
Original
Ours
224 224
74.5742
41.8925
128
128
65.2988
57.5542
448 448
19.9039
19.4923
224
224
23.0483
22.7971
768 768
29.5974
28,1372
384 384
20.5466
20.3842
512 288
22.6249
21.3877
256 144
33.8559
33,0142
512 384
20.2315
19.8631
256 *
192
24.2546
23.9346
Table 2: CLIP scores (1) for Stable Diffusion and Latent Diffusion in different resolution settings. Stable Diffusion scores 0.3158 for default 5122 and Latent Diffusion scores 0.3153 for default 256
Stable Diffusion - 512
Latent Diffusion 256
Resolution
Original
Ours
Resolution
Original
Ours
224 * 224
0,2553
0.2764
128 * 128
0.2679
0.2747
448 448
0.3176
0.3180
224
224
0.3094
0.3096
768 768
0.3142
0.3152
384 * 384
0.3148
0.3156
512 288
0.3047
0.3066
256 144
0.2884
0.2890
512 384
0.3134
0.3139
256 192
0.3070
0.3075
Table 3: Results of human evaluation. We report the score of consistency (1) for the text-to-image results of Stable Diffusion in different resolution settings.
Resolution 224 * 224
448
448
768 * 768
512 * 288
512 * #84
Original
5.13
6.45
6.74
6.11
6.75
Ours
6.77
7.12
7.56
7.42
6.98
Quantitative comparison. We compare the performance of the proposed scaling factor against the scaling factor X = 1/vd for Stable Diffusion (37 and Latent Diffusion (37 in multiple squared resolution settings (small, medium and large sizes according to their default training sizes). We evaluate them with the metric of Fréchet Inception Distance (FID) [13, which measures image quality and diversity. As shown in Table [ our scaling factor universally improves FID for both Stable Diffusion and Latent Diffusion in all resolution settings, especially for the images with small resolution (a significant improvement from 74.6 to 41.9 for Stable Diffusion on the resolution 224 x 224). It is worth noting that the proposed method needs no additional training and achieves these in a plug-and-plas manner with trivial complexity.
Besides FID, we also evaluate the degree of semantic alignment between text prompts and the corresponding synthesized images with CLIP scores (35). which utilizes contrastive learning to establish connections between visual concepts and natural language concepts. As shown in Table 2 our scaling factor achieves better CLIP scores than the original one in all resolution settings. It shows that our scaling factor performs better in the alignment of text prompts and the synthesized images.
Additionally, we present the results of user study Table 3 We conduct a text-based pairwise preference test. That is, for given sentence, we pair the image from our model with the synthesized image from the baseline model and ask 45 annotators in total to give each synthesized image rating score for consistency. For each human annotators, we pay S15 for effective completeness of user study evaluation. We observe that users rate higher for our method, which suggests that the synthesized images from our method are more contextually coherent with texts than the baseline. Besides, with the refinement from the proposed scaling factor, the generated contents from our model able to convey more natural and informative objects.

---

A ployground with a dog
Peopre on coaret beach
An octopus in the sea
Aquiere cotinn bug
Strvet with curs deylight
Town Ford Min Round
Sunglosses in Brown
Stable Diffusion
Ours
doing
Anciion
for graves
Low Stesl Yoe Work Shore
Chise! Bright green
Siiting duck
Stable Diffusion
Ours
A cup of milk toblee
Cut iving the groouu
Train on the railvayy
A girl millin
Wedding charch
Angry pando the
Stable Diffusion
Ours
Qualitative comparison on the scale factors for the resolution 224 224. The original scaling factor misses out content or roughly depicts the objects in prompts while our scaling factor manages to synthesize visual concepts in high fidelity and better illumination. Please zoom for better visual effect.
Figure 3:
Qualitative comparison. In Figures and we show some visualization results for the qualitative comparison between scaling factors with Stable Diffusion. Paired images are synthesized with the same seeds and text prompts. Please zoom in for the best visual effect. For Figure 3 we demonstrate results under the proposed and the original scaling factors for Stable Diffusion with 224 x 224 resolution, which is lower than the default training resolution 512 x 512 of Stable Diffusion. We observe that the original Stable Diffusion setting, objects are either missed out (e.g., absent dog, people and octopus) or roughly depicted (c.g., crudely rendered squirrel, cars and person with sunglasses). Our interpretation is that for lower resolutions, according Eq. (5). the model is synthesizing images with less entropy during the inference phase when compared to its training phase, which implies that the model is suffering from deprived information. In this way, the model. has more difficulty in generating complex objects and cannot generate attractive images. As for the results generated by the proposed scaling factor, we show that images are synthesized with better alignments with prompts (e.g., clear presence of dog, people and octopus) and objects are generated more vividly (e.g., realistic squirrel, cars and person with sunglasses). Note that our scaling factor also performs better in the illumination and shading because of the utilization of more aggregated
7

---

BECOMG MAGR
MORG I TALE SIIY
Stable
Diffusion
BLABIG BIBLL SRIYY
Ours
BCCONN BOOUMIEY
Figure 4: Qualitative comparison on the scale factors for the resolution 768 768. The original scaling factor generates objects in repeated and messy patterns while our scaling factor manages to spatially arrange the visual concepts in clear order. Please zoom in for better visual effect.
7,0
6.0
6.0
Eatropy (bits)
5.0
Entrooy
5.0
4.0
4.0-
3,0
2.0
6.0
7.0
8.0
9.0
10.0
6.9
6.8
7.0
7.8
80
logN
logN
Figure 5: Analysis results on the average attention entropy with respect to the number of tokens for Stable Diffusion model (left) and latent diffusion model (right). N refers to the number of tokens. As shown in the figure, average attention entropy has a linear relationship with the logarithm of N, which confirms the effectiveness of our asymptotic equality in (q.
and contextualized information with higher entropy. Considering its flexibility, our method might offer valuable insights into the fields of inpainting 28 56 and compositiona scene generation [59].
For Figure 4 we demonstrate the results of Stable Diffusion under the proposed and the original scaling factors with the 768 x 768 resolution, which is higher than the default training resolution 512 x 512 of Stable Diffusion. Different from the rough objects in Figure 3 we note that objects are depicted with overwhelming details in a messy and disordered manner (e.g., repeated hands, extended vehicle and messy seals). The reason is that for higher resolutions, according to (5) the model are bearing more entropy than it does in the training period. As a result, tokens are attending to numerous but redundant global information and the model depicts objects with burdensome details, leading to the patterns observed. For the results by our scaling factor, we note that the repeated patterns greatly alleviated (e.g., pair of normal hands and regular vehicle) and synthesized objects are spatially arranged in order without losing the details (three seals well depicted in order).
4.2 Analysis
In this section, we conduct comprehensive analysis of the practical implications of our theoretical findings. This analysis is of great importance due to the nature of our fundamental theoretical discov- ery outlined in Eq. As an asymptotic equality, it might exhibit limitations practical scenarios where the token quantity, denoted as N, falls below a certain threshold. Thus, it becomes imperative to assess the feasibility and the applicability of our theoretical finding under such cireumstances. Subsequently, we delve into an in-depth examination of the pivotal role played by the scaling factor
8

---

Resolution: 224 224
Resolution: 768
6.00
9.00
8.01
Eatropy (bits)
4.001
Entropy (hits)
7.00-
DM
Ours
6.00
2,00-
0.00
4.00
10
1.5
30
10
20
Layers_ID ID
Layers
Figure 6: Analysis results on the average attention entropy with respect to two scale factors for two resolution. Layer ID refers to the index number of self-attention layers in the model and DM stands for original diffusion model. The figure illustrate that the proposed scaling factor mitigates the fluctuation of attention entropy.
in mitigating the fluctuations within the attention entropy, as presented in Section 3.2 This rigorous analysis serves to substantiate and validate the insights elucidated in the aforementioned section. To support our claims, we present visually intuitive representations of the empirical results, which unequivocally demonstrate the alignment between our statements and the experimental outcomes.
The relationship between the attention entropy and the token number. We conduct an experi- ment text-t--image tasks to investigate the relation between the entropy and the token number N, which is statistically connected Eq. (5). particular, we synthesize images with various resolutions (including 224 3842, 448², 5122 and 768 for Stable Diffusion and 128 2242, 256 and 384² for Latent Diffusion model) and calculate their attention entropy during the inference period. We average the results across attention heads and layers for 5K samples on each resolution. Figure FigureBjillustreen iillusttase that the average attention entropy has a linear relationship with the logarithm of the quantity of tokens for both diffusion models, which accords with their asymptotic relationship proposed in E5. The visualized results provide the empirical evidence that substantiates the efficacy of our theoretical findings and the feasible extensions for those resolutions close to the mentioned resolutions.
The role of scaling factor for stabilizing the attention entropy. We conduct an experiment on the mitigating effect the proposed scaling factor onto the fluctuated attention entropy. Specifically, with two scaling factors, we synthesize images with lower and a higher resolution (including 224 and 768 for Stable Diffusion) and record their attention entropy for each attention layers. We average the results across attention heads for 5K samples. As demonstrated in the Figure 6 the proposed scaling factor does contribute to an increase in the synthesis process for the resolution 224 and a decrease in that for the resolution 7682, which accords with our motivation in Section 3.2 and shows the efficacy our scaling factor in mitigating the fluctuation of the attention entropy.
4.3 Difference with other candidate methods
In this section, we discuss upon the difference between the proposed method and other candidate meth- ods for variable-sized image synthesis, including up/down-samplin methods. While all the mentioned methods share the same goal to synthesis images of variable sizes, there are three distinctive features setting our method apart from the rest.
methods and super-resse reseutrcsolution
Different aspect ratio. Other candidate methods do not support diffusion models generate new images with a different aspect ratio. In comparison, our method could improve image synthesis in different aspect ratio, validated by both qualitative and quantitative experiments in Section 4.1
Richness of visual information (important for higher resolutions). When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Under this circumstance, our method augments semantic information by enabling models to deal with more tokens in an adaptive manner, which enables images to possess more extensive range of content. As shown in Figure 7 our method generates images with more

---

Diffusion
Up-sampled
Dirfusion
Ours 1024 x 1024
512 x 512
1024 x 1024
1024 x 1024
Undoption Internationale.
WINE AND
Everyday Mango Poster on wall.
EVEVRYRAY FFNEJM ANO
Out of the Fog-
Figure 7: Qualitative results to illustrate the difference in visual information richness between up- sampled methods, the original Stable Diffusion and the proposed method. Note that method (right) introduces more visual information with a different level of granularity compared with up-sampling and super-resolution methods (left). Additionally, our method could better deal with repetitively disordered pattern emerging in the original Stable Diffusion model (middle).
richness and expressiveness in semantics. In contrast, super-resolution and other methods scale up the original images and focus better image clarity and finer details instead of richer semantics, introducing visual information at different granularity level compared with the proposed method.
Time cost and memory usage (important for lower resolutions). For diffusion models adapted by our method, their time cost and spatial usage become proportional the generation resolution, while down-sampling methods are constantly constrained to the fixed cost brought by training resolutions. As result, our method could efficiently enable low resolution image synthesis especially on portable devices, which has high demand for both time cost and memory usage other than image fidelity. For more quantitative results to support this, please refer to the Supplementary Materials.
5 Conclusion
In this work, delves into perspective of adapting diffusion models to effectively synthesize images of varying sizes with superior visual quality, based on the concept of entropy. We establish a statistical connection between the number of tokens and the entropy of attention maps. Utilizing this connection, we give interpretations to the defective patterns we observe and propose a novel scaling factor for visual attention layers to better handle sequences of varying lengths. The experimental results demonstrate that our scaling factor effectively enhances both quantitative and qualitative scores for text-to-image of varying sizes in training-free manner, while stabilizing the entropy of attention maps for variable token numbers. Moreover, we provide extensive analysis and discussion for the efficiency and the significance of our method.
Limitations One limitation of this study pertains to the metrics used for quantitative evaluation.
While these metrics primarily evaluate the quality of the generated content, there is a lack of methodology for evaluating the fidelity of images for different resolutions. The same issue applies for the evaluation of repetitively disordered pattern, which is in part compensated by qualitative scores.
10

---

Acknowledgment
This work was supported in part by the National Key R&D Program of China (No.202IZDD) 2200 12803), the National Natural Science Foundation of China (No.62176060,
STCSM project 00.222511105000 Shanghai Municipal Science and Technology Major Project (2021 SHZDDZX000)))) UniDT's Cognitive Computing and Few Shot Learning Project, and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions Higher Learning.
References
[1] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.0884 2020.
[2] Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. Entropy-based attention regularization frees unintended bias mitigation from lists. arXiv preprint wxXiv:2203.09/92 2022.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 1rXiv:1409.0473, 2014.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-doeument transformer. arXiv preprint / 2020.
[5] Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pages 170-188. Springer, 2022.
[6] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages
2021.
[7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint 11Xi::1904.10009 2019.
[8] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11422 2022.
[9] Tri Dao, Beidi Chen, Kaizhao I Liang, Jiaming Yang. Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2112.00029 2021.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794, 2021.
[1]] Hamidreza Ghader and Christof Monz. What does attention in neural machine translation pay attention to? arXiv preprint arXiv:1710.03348, 2017.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfrr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prone image editing with cross attention control. arXiv preprint #XXv:2208.00626 2022.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230:3 2022.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 18684006851, 2020.

---

[16] Jonathan Ho. Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23(47):1-33, 2022.
[17] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.00 2244442200 2022.
[18] Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang. Tieniu Tan, and Jian Sun. Meta-sr: A arrbiirr network for super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1575-1584, 2019.
[19] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision, pages 1501-1510, 2017.
[20] DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block- recurrent transformers. arXiv preprint arXiv:2203.07852 2022.
[21] Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Style spectroscope: Improve interpretability and controllability through fourier analysis. arXiv preprint arXiv:2208.061 04040, 2022.
[22] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint 2rXiv:2066.00344 2022.
[23] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprini vrxii:2001.04455/ 2020.
[24] Krzysztof Lis, Matthias Rottmann, Sina Honari, Pascal Fua, and Mathieu Salzmann. Attentropy: Segmenting unknown objects in complex scenes using the spatial attention entropy of semantic segmentation transformers. arXiv preprint arXiv:22 14397, 2022.
[25] Peter J Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv: 1801. /0/98 2018.
[26] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[27] Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. A closed-form solution to universal style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5952-5961, 2019.
[28] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461-11471, 2022.
[29] Ye Ma, Jin Ma, Min Zhou, Quan Chen, Tiezheng Ge, Yuning Jiang, and Tong Lin. Boosting image outpainting with semantic layout prediction. arXiv preprint arXiv:2 10.09267 2021.
[30] Lovish Madaan, Srinadh Bhojanapalli, Himanshu Jain, and Prateek Jain. Treeformer: Dense gradient trees for efficient attention computation. arXiv preprint arXiv:2208.09011 2022.
[31] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference Learning Representations, 2021.
[32] Byung-Doh Oh and William Schuler. Entropy-and distance-based predictors from gpt-2 attention patterns predict reading times over and above gpt-2 surprisal. arXiv preprint arXiv:2212.11185 2022.
[33] Adam Pardyl, Grzegorz Rypešé, Grzegorz Kurzejamski, Bartosz Zieliński, and Tomasz Trzciński. Active visual exploration based on attention-map entropy. arXiv preprint 2:X1v:2303.06457,200 2023.
12

---

[34] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. International conference on machine learning, pages 6055-4064 PMLR, 2018.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 1748-8765 PMLR, 2021.
[36] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25, 2022.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022.
[38] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content- content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 2021.
[39] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: amagee diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[40] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic eext-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-366 -94994, 2022.
[41] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image super-resolutie via iterative refinement. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b:
An open large-scale dataset for training next generation image-text models. arXiv preprint 20. 2022.
[43] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fitered 400 million image-text pairs. arXiv preprint 1rXiv:2111.022 14, 2021.
[44] C. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379-42) 1948.
[45] Jascha Sohl-Dich Dicksteii, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[47] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 20t.:::20111145555 2020.
[48] Jianlin Su. Revisiting attention scale operation from the invariance of entropy, Dec 2021, [Blog post]. Retrieved from https:t/ttx:ut fm. wtt.aarchivee//e/
[49] Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention.
In International Conference on Machine Learning, pages 1438-9447. PMLR, 2020.
13

---

[50] Piotr Teterwak, Aaron Sarna, Dilip Krishnan. Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10521- 10530, 2019.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665-21674, 2020.
[53] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-195 1914, 2021.
[54] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks, In Proceed- ings of the European conference on computer vision (ECCV) workshops, pages 0-0, 2018.
[55] Yuxin Wang. Chu Tau Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. What dense graph do you need for self-attention? In International Conference Machine Learning, pages
PMLR, 2022.
[56] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034, 2022.
[57] Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, and Yashar Mehdad. Simple local attentions remain competitive for long-context tasks. arXiv preprint arXiv: /2.07210, 2021.
[58] Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10561-10570, 2019.
[59] Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[60] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontanon, Philipp Pham, Anirudh Ravula, Qifan Wang. Li Yang, at. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283- 2020.
14


Reviews:
[{"review_id": "ynSzczHDVJ", "reviewer": "Reviewer_p6J2", "summary": "In this paper, the authors propose a new scaling factor for attention based text-to-image generative models in order to handle variable sized generations. The authors establish the relationship between attention entropy and token size and use this newly found relationship to design a scaling factor that takes into account the image resolution. The authors also empirically show the effectiveness of this scaling factor by comparing FIDs, CLIP scores and qualitative examples.", "questions": "1. Related to Weakness (1), Section 3.2 is still a little bit confusing to me, can the authors explain more on the connection between Equation 5, 6 and 7? I.e. How did the authors come up with Equation 6 and 7?\n2. Related to Weakness (2), can the authors compare their method with synthesis in fixed pretrained resolution and perform super-resolution/downsampling? Can authors also compare to more dramatic resolution changes (eg. 512 => 128 or 512 => 1024)?\n3. Related to Weakness (3), can the authors compare their method with MultiDiffusion?\n4. Is FID appropriate for evaluating wrt different resolutions? Since it is trained on fixed resolution images. And how did the authors select the reference/ground truth images for FID evaluation? Did they uniformly rescale to the desired resolution or did they subsampled based on the native resolution? It may have a difference in the evaluation results.\n5. Although CLIP score is a reasonable indicator, I don’t think it is a perfect metric especially when object count is involved. Have the authors considered using off-the-shelf object detectors as one of the methods to quantitatively measure the performance?", "limitations": "Related to Weakness (4), the compensation for human annotators is not specified in the paper.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed scaling factor is very simple and easy to implement. It does not require model architecture modifications or any training of the pretrained model. The qualitative results also look very promising. Therefore I think this can be easily applied in many existing models. Overall the paper is well organized and well written.", "weaknesses": "1. The main formula (Equation 6 and 7) is not very well explained. There are a lot of conjectures and approximations without justifications.\n2. The authors did not compare with synthesis in fixed resolution and then performing super-resolution/downsampling. Since the resolution experimented in this paper is not very far away from the pretrained resolution, it is highly likely that super-resolution/downsampling may work just as well.\n3. The authors should also compare their method with other machine learning based methods such as MultiDiffusion (https://multidiffusion.github.io/)\n4. The compensation for human annotator is not mentioned\n5. Typo in Equation 1,3: j is used in both the numerator and the denominator of Aij", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: More explanation on Equation 5, 6 and 7.\n\n**A1**: In Equation 5, we derive the static relationship between attention entropy $A_{i}$ and token number $N$, i.e. $Ent(A_{i}) = \\log N - \\frac{1}{2} \\lambda^{2} C + O(1)$, where $\\lambda$ is the scaling factor and $C$ is a constant number unrelated with token number $N$. Recap that our goal is to control the fluctuating entropy $Ent(A_{i})$. Considering the form in Equation 5, we consequently set $\\lambda$ with a granularity of squared logarithm (i.e., $\\lambda = \\alpha \\sqrt{\\log N} $ ), which is Equation 6 in the paper and $\\alpha$ denotes a newly introduced hyper-parameter. Note that during training periods, token number $N$ is set as a constant number $T$ for training with $\\lambda$ fixed as $1 / \\sqrt{d}$ , which indicates that $\\lambda$ comes to $1 / \\sqrt{d}$  when $N = T$. In this way, we have $\\lambda = \\alpha \\sqrt{\\log T} \\approx 1 / \\sqrt{d}$ and we could have an approximate analytical solution for the introduced hyper-parameter $\\alpha \\approx 1 / (\\sqrt{d\\log T})$. By substituting this into Equation 6, we have $\\lambda \\approx \\sqrt{\\frac{\\log_T N}{d}}$, which is Equation 7. \n\nIn brief, \n\n- Considering the granularity of $\\lambda$ from Equation 5, we get Equation 6.\n- Combining Equation 6 with the condition that $\\lambda \\approx 1 / \\sqrt{d}$ when $N = T$, we get Equation 7.\n\n**Q2**: Comparison to super-resolution/downsampling methods.\n\n**A2**: We would like to point out the advantages of our method against other methods in three key aspects.\n\n1. **Better Support in Different Aspect Ratio**. Other methods do not support generating new images with a different aspect ratio from the original images with fixed resolutions. In comparison, our method could improve image generation in different aspect ratio, which is validated by both qualitative and quantitative experiments in our paper.\n\n2. **Richness of Visual Information (More Important for Higher Resolutions)** . When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Our method could introduce more information by enabling models to deal with more tokens, while super-resolution and other methods do not contribute to the richness of visual information by simply scaling up the original images. \n\n    To illustrate the non-trivial difference in the richness, we have provided examples in Figure 1 in the PDF page. \n\n3. **Time Cost and Memory Usage (More Important for Lower Resolutions)** . For diffusion models adapted by our method, their time cost and spatial usage become proportional to the generation resolution, while downsampling methods are constantly constrained to the fixed cost brought by training resolutions. In this way, our method could efficiently enables low resolution image synthesis especially on portable devices, which have a high demand for both time cost and memory usage other than image fidelity. \n\n    In Table 1, we present FID, memory usage and time cost of stable diffusion with different methods in resolution 224. For low resolution, our method achieves better performance in memory usage and time cost, with partial trade-off in fidelity. Noticing those fine qualitative results in Figure 3 in the paper, we believe the trade-off is acceptable considering the tackling scenario of low resolution image synthesis. \n\n| Resolution                        | GPU Memory (MiB) | FID     | Average Time (s) |\n| --------------------------------- | ---------------- | ------- | ---------------- |\n| 224 * 224 (Original)              | 7064             | 74.5742 | 5.6268           |\n| 224 * 224 (Ours)                  | 7064             | 41.8925 | 5.6274           |\n| 512 * 512 and Downsampling to 224 | 9324             | 21.8415 | 17.4457          |\n\nTable 1: FID, memory usage and time cost of stable diffusion with different methods with resolution $224^{2}$. For more results, please refer to Table 1 in other rebuttals due to character limits.\n\n**Q3**: Comparison with MultiDiffusion.\n\n**A3**: We compare our method with MultiDiffusion and report corresponding FID(4K), memory usage and time cost in Table 2. As shown in Table 2, our method outperforms MultiDiffusion in both resolutions except for the GPU Memory with high resolution.\n\n| Resolution       | GPU Memory (MiB) | FID (4K) | Average Time (s) |\n| ---------------- | ---------------- | -------- | ---------------- |\n| 224 * 224 (Ours) | 7064             | 50.3515  | 5.6274           |\n| 224 * 224 (MD)   | 7061             | 154.6925 | 31.7680          |\n| 768 * 768 (Ours) | 19797            | 28.1372  | 36.7140          |\n| 768 * 768 (MD)   | 9906             | 40.9270  | 122.2485         |\n\nTable 2: FID (4K samples), memory usage and time cost of diffusions with resolution $224^{2}$ and $768^{2}$.\n\n**Q4**: Limits on the FID metric.\n\n**A4**: In this paper we only use FID for the comparison between images with the same resolution (w/ and w/o our method) to validate the efficacy of our method. Thus, we believe the comparison is fair and credible. Specifically, we uniformly rescale the reference images to desired resolution for FID. In addition, we have discussed about the limits of FID in the Limitation Section and supported our claims with additional qualitative experiments and user study evaluation.\n\n**Q5**: Using off-the-shelf object detectors.\n\n**A5**: Please note that repetitive patterns in images are not expected to be detected by object detectors since the repetitiveness exhibits in the form of texture/composition rather than countable objects (please refer to Figure 5 in the Supplementary Materials).  Thus, we think using object detectors might not help. \n\n**Q6**: Typos and unspecified compensation.\n\n**A6**: We have revised our script by changing j in the numerator into j' and adding \"For each human annotators, we pay $15 for effective completeness of user study evaluation\" in Section 4.1, Line #213."}, {"review_id": "SLLB4wy7sG", "reviewer": "Reviewer_wERV", "summary": "This work adapts a pre-trained Stable Diffusion model for variable-resolution image generation. Since Stable Diffusion is trained on a fixed image resolution, naively varying the output size results in abnormal patterns in the images. This paper tracks the problem down to self-attention weights in the denoiser network, and identifies the so-called attention entropy as the root cause by proving that attention entropy is fundamentally correlated with image resolution. Drawing on this insight, the paper introduces resolution-dependent scaling to self-attentions to calibrate attention entropy throughout the generative process. The effectiveness of the proposed method is validated through extensive experiments.", "questions": "Please see the section above for questions.", "limitations": "The paper discussed the limitations and societal impact of the method.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The method is training-free. It works by modifying the generative process of a pre-trained diffusion model at inference time.\n\n- The method reveals a strong connection between self-attention entropy and image quality. It builds on the key finding that attention entropy is correlated with image resolution, and artifacts in low / high-resolution images can be attributed to mis-calibrated attention entropy. It thus attempts to calibrate attention entropy across various image resolutions using a resolution-dependent scaling factor. Turning theoretical insights into a simple, actionable solution is a key strength of the paper.\n\n- The effectiveness of the method is validated by qualitative, quantitative and user study results. The improvement over the Stable Diffusion baseline seems quite consistent.", "weaknesses": "- A few simple baselines are missing. For example, one can simply down-sample a 512x512 image to reach a lower resolution. Similarly, one can generate a 512x512 image and subsequently up-sample it using an off-the-shelf super-resolution model. In fact, Stable Diffusion supports super-resolution and uneven aspect ratios with community effort (check out AUTOMATIC1111). To justify the main contribution of the paper (i.e., improved generation quality of variable-sized images), it is important to show that the proposed method performs equally well, or even better, compared to these simple baselines.\n\n- Arguably, generating high-resolution images is of greater interest to the community. To this end, it would be interesting to probe the limit of the proposed method. That is, what is the highest resolution it can handle without introducing noticeable artifacts? Most high-resolution images in the paper are 768x768. There is one image in the supplement at the resolution of 1024x1024. The proposed method will generate more impact if it can further grow the image resolution.\n\n- According to Figure 5 (left), attention entropy is up by 1 bit as image resolution grows from 512 to 768. However, Figure 6 (right) seems to suggest that applying the scaling factor only reduces entropy by a very small amount. My early impression is that the scaling factor aims to restore entropy to the level of 512x512 images. I am curious about what causes this discrepancy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your careful and valuable comments. We will explain your concerns point by point.\n\n**Q1**: Comparison to baselines.\n\n**A1**: We would like to point out the advantages of our method (modifying fixed-resolution models) against other methods (e.g. cascaded diffusion models and LIIF) in three key aspects.\n\n1. **Improvement in Different Aspect Ratio**. Super-resolution and downsampling methods do not support generating new images with a different aspect ratio from the original images with fixed resolutions. In comparison, our method could improve image generation in different aspect ratio, which is validated by both qualitative and quantitative experiments in our paper. For AUTOMATIC1111, the aspect ratios are passed directly into Stable Diffusion (like the original Stable Diffusion in our paper, which also support uneven aspect ratios) without any in-depth optimization.\n\n2. **Richness of Visual Information (More Important for Higher Resolutions)** . When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Our method could introduce more information by enabling models to deal with more tokens, while super-resolution and other methods do not contribute to the richness of visual information by simply scaling up the original images. \n\n    To illustrate the non-trivial difference in the richness, we have provided several examples in Figure 1 in the PDF page. \n\n3. **Time Cost and Memory Usage (More Important for Lower Resolutions)** . For diffusion models adapted by our method, their time cost and spatial usage become proportional to the generation resolution, while downsampling methods are constantly constrained to the fixed cost brought by training resolutions. In this way, our method could efficiently enables low resolution image synthesis especially on portable devices, which have a high demand for both time cost and memory usage other than image fidelity. \n\n    In Table 1, we present FID, memory usage and time cost of stable diffusion with different methods in various resolutions ($128^{2}, 224^{2}, 768^{2}, 1024^{2}$). For low resolution, our method achieves better performance in memory usage and time cost, with partial trade-off in fidelity. Noticing those fine qualitative results in Figure 3 in the paper, we believe the trade-off is acceptable considering the tackling scenario of low resolution image synthesis. \n\n| Resolution                       | GPU Memory (MiB) | FID      | Average Time (s) |\n| -------------------------------- | ---------------- | -------- | ---------------- |\n| 224 * 224 (Original)             | 7064             | 74.5742  | 5.6268           |\n| 224 * 224 (Ours)                 | 7064             | 41.8925  | 5.6274           |\n| 512 * 512 and Resize to 224      | 9324             | 21.8415  | 17.4457          |\n|                                  |                  |          |                  |\n| 768*768(Original)                | 19797            | 29.5974  | 36.7137          |\n| 768*768(Ours)                    | 19797            | 28.1372  | 36.7140          |\n| 512*512 and Resize to 768 | 9948             | 20.8712  | 18.5286          |\n|                                  |                  |          |                  |\n| 128 * 128 (Original)             | 6867             | 191.5447 | 3.4953           |\n| 128 * 128 (Ours)                 | 6867             | 127.2798 | 3.4955           |\n| 512 * 512 and Resize to 128      | 9324             | 36.0742  | 17.4377          |\n|                                  |                  |          |                  |\n| 1024 * 1024 (Original)           | 39977            | 50.5425  | 62.2414          |\n| 1024 * 1024 (Ours)               | 39978            | 43.1167  | 62.2359          |\n| 512 * 512 and Resize to 1024     | 12043            | 20.8235  | 20.8316          |\n\nTable 1: FID, memory usage and time cost of stable diffusion with different methods with different resolutions.\n\nIn brief, our method does better in supporting uneven different aspect ratio and adapts diffusion models with moderate fidelity trade-off for meeting important requirements both in high resolutions and low resolutions.\n\n**Q2**: The range limit of proposed method.\n\n**A2**: We have provided more 1024x1024 generated images in the PDF page, which show the outperforming performance and information richness of our method in synthesis. To be more specific, our method could improve FID by nearly 15% at the resolution of 1024x1024. We think the limit might depend on specific semantics (e.g., human faces) since we observe that Stable Diffusion sometimes could not successfully generate these semantics (e.g., human faces) even at the resolution of 512x512. Additionally, the proposed method could also get implemented onto other diffusion models (e.g. diffusion models trained on the resolution of 2048x2048 and get expanded to 4096x4096). \n\n**Q3**: Changing attention entropy. \n\n**A3**: We think the attention entropy should not be fixed because the information of images would naturally change in accordance with resolutions and the attention entropy would inevitably fluctuate.  Thus, the goal of our method in the paper is to \"alleviate the fluctuation\" instead of fixation."}, {"review_id": "0LWB8MOvsX", "reviewer": "Reviewer_QLpB", "summary": "This paper analyzes the issues of using a fixed-resolution diffusion model to generate varied-size images and proposes a scaling factor to stable the attention entropy which remedies the issue. The method is evaluated on text-to-image models when the inference resolution is moderately different than the model resolution and the results show the effectiveness of the proposed method compairing to directly using fixed-resolution diffusion model for varied-size synthesis.", "questions": "See weaknesses. It is not clear to me that, if a diffusion model only see faces at resolution 256 during training, how can its UNet synthesize a face at resolution 2048 with its kernels (trained for resolution 256) without repetitive patterns? Does the proposed method address this issue?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. The motivation is clearly explained and it is with some theoretical analysis.\n2. The proposed method is simple to implement, which does not require costly large scale training and could help varied-size synthesis with a fixed-resolution diffusion model.\n3. The effectiveness of changing the scale factor in attention for varied-size synthesis is an interesting observation.", "weaknesses": "1. Synthesizing images at different resolutions is an important research problem, but it is not very clear that why a fixed-resoltion pre-trained model should be used for this task (which is the main baseline in this work). To achieve this goal, there are many other candidate methods: (1) using diffusion-based (e.g. cascaded diffusion models) or GAN-based super-resolution method to upsample the output to some resolution of powers of 2, then downsample it to the query resolution. (2) using a any-scale super-resolution network (e.g. LIIF) to upsample the network result. It is not clear that why modifying fixed-resolution model is an important research topic if it's not shown to outperform other candidate methods.\n2. The test resolutions are still in a small changing range compared to the model resolution (about x0.5\\~x1.5), and the FID on x0.5 scale suggests that the quality could be significantly decreased when changing the scale to be different than training scale. Does this suggest that the proposed method does not work for a wider range of scales (for example, any-scale SR works typically evaluates x1\\~x4, or even extrapolates to x20\\~30)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments. We will explain your concerns point by point.\n\n**Q1**: Comparison to candidate methods.\n\n**A1**: We would like to point out the advantages of our method (modifying fixed-resolution models) against other methods (e.g. cascaded diffusion models and LIIF) in three key aspects.\n\n1. **Improvement in Different Aspect Ratio**. Other methods do not support generating new images with a different aspect ratio from the original images with fixed resolutions. In comparison, our method could improve image generation in different aspect ratio, which is validated by both qualitative and quantitative experiments in our paper.\n\n2. **Richness of Visual Information (More Important for Higher Resolutions)** . When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Our method could introduce more information by enabling models to deal with more tokens, while super-resolution and other methods do not contribute to the richness of visual information by simply scaling up the original images. \n\n    To illustrate the non-trivial difference in the richness, we have provided several examples in Figure 1 in the PDF page. \n\n3. **Time Cost and Memory Usage (More Important for Lower Resolutions)** . For diffusion models adapted by our method, their time cost and spatial usage become proportional to the generation resolution, while downsampling methods are constantly constrained to the fixed cost brought by training resolutions. In this way, our method could efficiently enables low resolution image synthesis especially on portable devices, which have a high demand for both time cost and memory usage other than image fidelity. \n\n    In Table 1, we present FID, memory usage and time cost of stable diffusion with different methods in various resolutions ($128^{2}, 224^{2}, 768^{2}, 1024^{2}$). For low resolution, our method achieves better performance in memory usage and time cost, with partial trade-off in fidelity. Noticing those fine qualitative results in Figure 3 in the paper, we believe the trade-off is acceptable considering the tackling scenario of low resolution image synthesis. \n\n| Resolution                       | GPU Memory (MiB) | FID      | Average Time (s) |\n| -------------------------------- | ---------------- | -------- | ---------------- |\n| 224 * 224 (Original)             | 7064             | 74.5742  | 5.6268           |\n| 224 * 224 (Ours)                 | 7064             | 41.8925  | 5.6274           |\n| 512 * 512 and Resize to 224      | 9324             | 21.8415  | 17.4457          |\n|                                  |                  |          |                  |\n| 768*768(Original)                | 19797            | 29.5974  | 36.7137          |\n| 768*768(Ours)                    | 19797            | 28.1372  | 36.7140          |\n| 512*512 and Resize (liif) to 768 | 9948             | 20.8712  | 18.5286          |\n|                                  |                  |          |                  |\n| 128 * 128 (Original)             | 6867             | 191.5447 | 3.4953           |\n| 128 * 128 (Ours)                 | 6867             | 127.2798 | 3.4955           |\n| 512 * 512 and Resize to 128      | 9324             | 36.0742  | 17.4377          |\n|                                  |                  |          |                  |\n| 1024 * 1024 (Original)           | 39977            | 50.5425  | 62.2414          |\n| 1024 * 1024 (Ours)               | 39978            | 43.1167  | 62.2359          |\n| 512 * 512 and Resize to 1024     | 12043            | 20.8235  | 20.8316          |\n\nTable 1: FID, memory usage and time cost of stable diffusion with different methods with different resolutions.\n\nIn brief, our method does better in supporting uneven different aspect ratio and adapts diffusion models with moderate fidelity trade-off for meeting important requirements both in high resolutions and low resolutions.\n\n**Q2**: The range limit of proposed method.\n\n**A2**:  We would like to state that our method does trade off FID because we do not have the same amount of calculation as the original resolution. However, as described in A1, our method could better meet important requirements for time cost and memory usage at lower resolutions or richness of information at high resolutions. We report the results of  stable diffusion with different methods in various resolutions ($128^{2}, 224^{2}, 768^{2}, 1024^{2}$) . Additionally, our method could be implemented with no training effort to other diffusion models (e.g., diffusion models trained on the larger resolution).  \n\n**Q3**: How can UNet synthesize a face at resolution 2048 with its kernels (trained for resolution 256) without repetitive patterns?\n\n**A3**: To be more specific, when the UNet is synthesizing a face at resolution 2048, our method aims to keep each token has the samilar behavior as it is at resolution 256, which means similar sized faces would be synthesized. To illustrate this, please refer to the Figure with prompt as \"L'adoption internationale\" in the PDF page."}, {"review_id": "68oZnQVW1N", "reviewer": "Reviewer_XLhR", "summary": "This paper proposes both an analysis and a contribution to fix a problem found during the analysis.\n\nThey start with the premise that diffusion models should be able to generate arbitrary size images, and training specialized models for each image size is too expensive, which is correct. Using diffusion models trained for square images would be much cheaper and easier.\nThey identify two key problems when using square models to generate arbitrary aspect ratio images: incomplete/inadequate objects and repetitive/disorganized patterns.\nIn order to improve performance all around (quality, prompt following, etc.) they study entropy in the generated image.\nSpecifically, they note that as entropy rises tokens attend to wider regions, and relate this phenomenon to the problems delineated above.\nFinally, they find that simply proposing a scaling factor mitigates many of these issues.", "questions": "1. I'm assuming time complexity does not vary with the new scaling factor?", "limitations": "1. Good limitation section. Metrics are a bit of an issue but there is a strong user eval in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea to add a scaling factor that counteracts entropy fluctuation in a training-free manner is really nice and the formulation is a principled approximation that is easily computable.\n2. The implementation is incredibly simple, a large plus.\n3. The work presents strong quantitative evaluations as well as a strong user evaluation that is commendable.\n4. The work presents a wide amount of qualitative samples that show improvement in very common problems for diffusion models such as double heads, double hands and other issues. Samples look good.", "weaknesses": "1. Hard to find many weaknesses, it's a strong paper that is well written, with a clear argument, clear analysis, clear solution that simply seems to work after both quantitative and user study evaluation (and with substantial qualitative samples showing edge cases).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We really appreciate your interest and support in our paper.\n\n**Q1**: Relation between time complexity and the new scaling factor.\n\n**A1**: Yes, it needs constant O(1) time complexity to calculate the new scaling factor."}]

Author rebuttal:
Thanks for all the reviewers and AC for your time and valuable comments!

This comment is followed by the PDF page with new figures to support our views in information richness.

Meta-review: All the reviewers lean towards the acceptance of the work. Reviewers appreciated the proposed training-free technique for variable size image generation as well as the technical insights. Reviewers raised several clarification questions and also raised concerns regarding some missing comparisons. Authors addressed several of these questions in their responses. The reviewers did raise some valuable concerns and suggestions that should be addressed in the final camera-ready version of the paper, which include adding the relevant rebuttal discussions and revisions in the main paper. The authors are encouraged to make the necessary changes to the best of their ability.
Decision: Accept (poster)