Title: Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis
Venue: NeurIPS 2023
Accepted: yes
Keywords: Text-to-Image Synthesis

Variable-Sized Image Synthesis

Entropy
Abstract: Diffusion models (DMs) have recently gained attention with state-of-the-art performance in text-to-image synthesis. Abiding by the tradition in deep learning, DMs are trained and evaluated on the images with fixed sizes. However, users are demanding for various images with specific sizes and various aspect ratio. This paper focuses on adapting text-to-image diffusion models to handle such variety while maintaining visual fidelity. First we observe that, during the synthesis, lower resolution images suffer from incomplete object portrayal, while higher resolution images exhibit repetitively disordered presentation. Next, we establish a statistical relationship indicating that attention entropy changes with token quantity, suggesting that models aggregate spatial information in proportion to image resolution. The subsequent interpretation on our observations is that objects are incompletely depicted due to limited spatial information for low resolutions, while repetitively disorganized presentation arises from redundant spatial information for high resolutions. From this perspective, we propose a scaling factor to alleviate the change of attention entropy and mitigate the defective pattern observed. Extensive experimental results validate the efficacy of the proposed scaling factor, enabling models to achieve better visual effects, image quality, and text alignment. Notably, these improvements are achieved without additional training or fine-tuning techniques.

Training-free Diffusion Model Adaptation for Variable-Sized Text-to-Image Synthesis

Abstract

Diffusion models (DMs) have recently gained attention with state-of-the-art performance in text-to-image synthesis. Abiding by the tradition in deep learning, DMs are trained and evaluated on the images with fixed sizes. However, users are demanding for various images with specific sizes and various aspect ratio. This paper focuses on adapting text-to-image diffusion models to handle such variety while maintaining visual fidelity. First we observe that, during the synthesis, lower resolution images suffer from incomplete object portrayal, while higher resolution images exhibit repetitively disordered presentation. Next, we establish a statistical relationship indicating that attention entropy changes with token quantity, suggesting that models aggregate spatial information in proportion to image resolution. The subsequent interpretation on our observations is that objects are incompletely depicted due to limited spatial information for low resolutions, while repetitively disorganized presentation arises from redundant spatial information for high resolutions. From this perspective, we propose a scaling factor to alleviate the change of attention entropy and mitigate the defective pattern observed. Extensive experimental results validate the efficacy of the proposed scaling factor, enabling models to achieve better visual effects, image quality, and text alignment. Notably, these improvements are achieved without additional training or fine-tuning techniques.

1 Introduction

Diffusion models have emerged as a powerful technique for image synthesis [10, 15, 46], achieving state-of-the-art performance in various applications [12, 28, 14]. Among them, text-to-image diffusion models have garnered significant attention and witnessed a surge in demand [40, 37, 36]. Traditionally, diffusion models have adhered to the typical deep learning approach of training and testing on images with predetermined sizes, which generally leads to high-quality results. They still exhibit a range of visual defects and diverse flaws when confronted with a novel synthesizing resolution (e.g., 512 2 in training while 224 2 in testing). However, real-world scenarios often demand the generation of images with diverse sizes and aspect ratios, necessitating models that can handle such variety with minimum loss in visual fidelity. The necessity becomes even more severe in the generation of large models. As the size of models continues to increase, the associated training costs experience a substantial surge, thereby posing financial challenges for average programmers and emerging startups, making it unfeasible for them to train specialized models tailored to their specific needs. Consequently, there is an urgent demand to explore methodologies that facilitate the full utilization of open-sourced models trained on fixed sizes.

In regard to this limitation, our first key observation reveals that most instances of poor performance could be attributed to two prevalent patterns: incomplete or inadequately represented objects and

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

(e) NYX Glitter Cream Palette color GCP02 Eden Brand 2 PACK

(f) BMW X5 3.0d Swap or Sell

(g) Colorful house against nature background

λ

λ

λ

λ

λ

λ

(d) Alps mountain

(a) A person on the Moon

λ

λ

λ

λ

(c) Forest mountain road

λ

λ

λ

λ

(b) Softail Deluxe

repetitively disordered presentations. We have included several examples of flawed synthesized images in Figure 1, designated by green icons. Note that smaller images (e.g., images (a), (b), (c) and (d)) conform to the first pattern, showcasing inadequately depicted objects. Conversely, larger images (e.g., images (e), (f) and (g)) exhibit the second pattern, generating disordered objects in a repetitive manner. This delineation, also observed in other problematic cases, allows us to formulate our second key observation, where lower resolution images are more vulnerable to the first pattern, while susceptibility to the second pattern increases in higher resolution images.

In this work, we tackle the challenge of adapting text-to-image diffusion models to proficiently synthesize images spanning a wide range of sizes, encompassing both low and high resolutions. Our goal is threefold: (1) achieve a higher level of fidelity regardless of synthesized image resolutions; (2) mitigate the abovementioned two patterns; and (3) augment the semantic alignment between the synthesized images and the text prompts.

To accomplish these, our approach centers around the concept of entropy, which measures the spatial granularity of information aggregation. Specifically, when the attention entropy rises, each token is attending to wider spatial information, otherwise the opposite. Bearing that in mind, we establish the statistical relationship between the resolution (the number of tokens) and the entropy of attention maps. Our finding signifies that the attention entropy varies in correspondence with the quantity of tokens, which implies that models are aggregating spatial information in proportion to resolutions. We then establish our key interpretations based on the proportionality. In particular, since narrower attention is paid during the synthesis of low resolution images, models encounter difficulties in representing complete objects with limited spatial information. Conversely, when synthesizing high resolution images, each token tends to conduct a wide spatial aggregation which results in a redundancy of analogous spatial information and the disordered presentations of repeated elements.

Based on the statistical relationship and interpretations, we propose a novel scaling factor specifically designed for mitigating the change of the entropy in visual attention layers. We conduct qualitative and quantitative experiments to validate the efficacy of our scaling factor, which synthesizes results of varied resolutions with better visual effect and achieves better scores in image quality and text alignment. It is worth noting that the improvement is implemented in a training-free manner by replacing a single scaling factor in the attention layers.

The contributions are summarized as follows:

We observe two distinct patterns of defects that low resolution images are prone to impaired object portrayal while repetitively disordered presentation exhibits in high resolution images.

We establish a statistical relationship between attention the entropy and the resolution, which interprets the patterns by ascribing them to the lack or the surplus of the spatial information.

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

We propose a scaling factor to efficiently improve variable-sized text-to-image synthesis in a training-free manner. We conduct various experiments and analysis to validate the efficacy.

2 Related Work

Image super-resolution and extrapolation. An alternative path to generate new images with different resolutions is post-processing synthesized images with image super-resolution and extrapolation methods. Super-resolution methods [18, 6, 53, 54] dedicate to exploiting the high frequency information of images and extending both the height and width of images by a fixed factor, which cannot change the aspect ratio of images. For extrapolation methods, they do have the ability to change the aspect ratio by extending images beyond their boundaries [50, 58, 29]. However, these methods are effective for signals that exhibit an intensive stationary component [5] (e.g., landscapes) while we would like to generate images with various scenes. Additionally, extrapolation methods might have discrepancy with diffusion models, resulting in inharmonious visual effect. Instead of adding more methods for post-processing and making the pipeline lengthy, we seek to synthesize high-fidelity images with diffusion models alone.

Diffusion models and attention mechanism. Diffusion models have emerged as a promising family of deep generative models with outstanding performance in image synthesis [10, 15, 22, 45, 46, 47]. They have significantly revolutionized computer vision research in multiple areas, including image editing [8, 12, 31], inpaintng [28, 39, 56], super-resolution [16, 41] and video synthesis [14, 17]. These models generate high-quality images by smoothly denoising a noisy image sampled from a standard Gaussian distribution through multiple iterations. A key component in diffusion models is the attention module. Previous works such as denoising diffusion probabilistic model [15], denoising diffusion implicit model [46], latent diffusion model [37] have demonstrated the capability empowered by the attention modules. Note that attention modules in diffusion models are dealing with thousands of visual tokens and one of the most well-known research concern of attention is how to lower its conventional quadratic complexity against the large token numbers. To deal with this, most works define a sparse attention pattern to restrict each query to attend to a limited subset of tokens exclusively. The sparse attention pattern could be either defined as fixed windows [23, 52, 49, 38, 30] or learned during training [25, 34, 7, 4, 1, 60, 26, 57, 55, 9, 20]. We draw upon the insights in these works for sparsity controlling and seek to efficiently adapt trained models for variable-sized data.

Attention entropy. Introduced in the information theory by [44], entropy serves as a measure for information and uncertainty. Different research works have introduced entropy to attention maps in various tasks, including semantic segmentation [24], text classification [2], text generation [48], active visual exploration [33] and interpretability analysis [32]. Among them, the work in text generation [48] closely aligns with ours on attention scaling, leveraging Transformers to accommodate sentences of variable lengths. We draw upon the innovations in the entropy of attention maps for information quantification and seek to exploit both of its practicability and interpretability.

3 Method

This section establishes the connection between the attention entropy and the token number in a statistical manner and presents a scaling factor replaced into diffusion model for variable-sized image synthesis. Our aim is to alleviate the fluctuation in attention entropy with varying token numbers.

3.1 Connection: the attention entropy and the token number

Diffusion models refer to probabilistic models that learn a data distribution by iteratively refining a normally distributed variable, which is equivalent to learn the reverse process of a fixed Markov Chain with a length of T . In image synthesis, the most efficacious models make use of a Transformerbased encoder-decoder architecture for the denoising step. This step can be understood as an evenly weighted chain of denoising autoencoders, which are trained to predict a filtered version of their input via cross-attention. Yet the scaling factor inside the attention module of diffusion models is so far an under-explored area of research. In this section, we focus on enhancing conditional diffusion models by adaptively determining the scaling factor of Transformer encoder-decoder backbones.

Let X ∈ R N × d denotes an image token sequence to an attention module, where N,d are the number of tokens and the token dimension, respectively. We then denote the key, query and value matrices as K = XW K , Q and V , where W K ∈ R d × d r is a learnable projection matrix and d r is the projection dimension. The attention layer ([3, 51]) computes Attention ( Q , K , V ) = AV with the attention map A calculated by the row-wise softmax function as follows:

where λ is a scaling factor, usually set as 1 / d in the widely-used Scaled Dot-Product Attention [51] and i, j are the row indices on the matrices Q and K , respectively. Following [2, 11], we calculate the attention entropy by treating the attention distribution of each token as a probability mass function of a discrete random variable. The attention entropy with respect to the i th row of A is defined as:

̸

where the attention entropy gets its maximum when A i,j = 1 /N for all j and gets its minimum when A i,j ′ = 1 and A i,j = 0 for any j = j ′ . It implies that the attention entropy measures the spatial granularity of information aggregation. A larger attention entropy suggests that wider contextual information is taken into account while a smaller entropy indicates only few tokens are weighed.

We now investigate the relation between the attention entropy and the token number. By substituting Eq.(1) into Eq.(2), we have:

where E j denotes the expectation upon the index j , and the last equality holds asymptotically when N gets larger. For the complete proof, please refer to the Supplementary Materials.

Given X as an encoded sequence from an image, each token X j could be assumed as a vector sampled from a multivariate Gaussian distribution N ( µ X , Σ X ) . This prevailing assumption is widely shared in the fields of image synthesis [13] and style transfer [19, 27, 21]. In this way, each K j = X j W K could be considered as a sample from the multivariate Gaussian distribution N ( µ K , Σ K ) = N ( µ X W K , ( W K ) ⊤ Σ X W K ) . In regard to K j , its dimension-wise linear combination y i = λ Q i K ⊤ j could be treated as a scalar random variable sampled from the univariate Gaussian distribution y i ∼ N ( µ i , σ 2 i ) = N ( λ Q i ( µ K ) ⊤ , λ 2 Q i Σ K Q ⊤ i ) . Under this circumstance, we dive into the expectations E j ( e λ Q i K ⊤ j ) and E j ( λ Q i K ⊤ j e λ Q i K ⊤ j ) in Eq.(3):

For the complete proof, please refer to the Supplementary Materials. By substituting Eq.(4) into Eq.(3), we have:

The observation is that the attention entropy varies correspondingly with N in a logarithm magnitude while the latter item σ 2 i = λ 2 Q i Σ K Q ⊤ i is not relevant with the token number N in the setting of images generation. This implies that each token intends to adaptively deal with wider contextual information for higher resolutions, and narrower contextual information for lower resolutions.

Interpretations on two observed patterns. Our key insight lies in the correlation between the fluctuating attention entropy and the emergence of two defective patterns in our observations. Specifically, when the model is synthesizing images with lower resolutions, the attention entropy experiences a statistical decline of log N , leading to a reduced attention on contextual information. Consequently, each token is processing discrete information in isolation and the model struggles to generate objects with intricate details and smooth transitions, giving rise to the first observed pattern.

<latexi sh

1_b

64="

DI

WrR+U

BwfV9SLM

kTO

H8

>A

C

n

c

g

F

3

d

X

oJ

mZE

0

Y

y

Q

N

/

j

7

v

2

p

u

K

z

P

5

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="Nn

PJ

q

8vHB

Zu/7

oA

S+

w

>

X

c

VDL

F

3U

2

5G

yC

I

Td

0Y

Q

O

k

E

z

W

9

p

fK

r

j

g

R

m

M

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

<latexi sh

1_b

64="

vgDIMP3

7RKnVS

Tq

m

8/

Y

>A

CBH

c

L

N

FJ

U

2

u

o

QdO

y

E

Z

0

+

5

zr

9

p

f

W

G

w

X

QK

V

QK

V

Denoising Step

=

√

1

d

=

√

log

T

N

d

λ

λ λ

ATT(Q,K,V) = softmax(

λ

(

QK

/latticetop

))

V

λ

Replace Scaling Factor

Conversely, in scenarios involving higher resolutions, the attention entropy exhibits an upward trend, accompanied by the utilization of overwhelming amounts of global information for synthesis. Consequently, each token is processing analogous information and the model generates repetitively disordered presentations, exemplifying the second pattern.

3.2 A scaling factor for mitigating entropy fluctuations

To alleviate the fluctuating entropy with respect to different token numbers N in the inference phase, considering the quadratic form in Eq.(5), we set the scaling hyper-parameter λ in the following form:

λ = c √ log N, (6) where c denotes a constant number. Note that when the token number N is set as the number of tokens during training (denoted as T ), λ should specialize to 1 / √ d for the best stablization of entropy (i.e., c √ log T ≈ 1 / √ d ). Thus, we have an approximate value of c and substitute it into Eq.(6):

Intuitively, if the model is dealing with a token number N > T , the proposed scaling factor would multiply the feature maps by a factor of squared log T N to maintain the rising entropy, and vice versa. Thus, the proposed scaling factor is implemented in a training-free manner, which enjoys better performances in synthesizing images of different resolutions. The illustration of our proposed method is shown in Figure 2.

4 Experimental Results

In this section, we evaluate our proposed method on text-to-image synthesis and analyse the efficacy of the proposed scaling factor. The quantitative and qualitative results demonstrate a better performance of our scaling factor. Please refer to the Supplemental Materials for detailed experimental results.

4.1 Text-to-image synthesis

Experimental setup. We explore the adapting ability of the proposed scaling factor for diffusion models in variable-sized text-to-image synthesis. In particular, we replace the scaling factor of selfattention layers within diffusion models and evaluate their performance without any training upon a subset of LAION-400M and LAION-5B dataset ([43, 42]), which contain over 400 million and 5.85 billion CLIP-filtered image-text pairs, respectively. We randomly choose text-image pairs out from each dataset and generate images corresponding to texts in multiple settings for evaluation. As for the diffusion models, we consider both Stable Diffusion [37] trained on LAION-5B and Latent Diffusion [37] trained on LAION-400M to test the performances. The former (which is set to synthesize images with a 512 × 512 resolution by default) maintains 4x longer sequences compared with the latter (with a default 256 × 256 resolution) across all the layers. For more evaluation results and details, please refer to the Supplementary Materials.

<latexi sh

1_b

64="R

TN

jK

gQ

o

cO

IEJ

AM+Y

>

B9H

VDLS

F

X7W

q

d

v

y

u

G5

8

k

m

/

3

2

Z

z

n

r

P

w

0

U

p

f

C

Quantitative comparison. We compare the performance of the proposed scaling factor against the scaling factor λ = 1 / √ d for Stable Diffusion [37] and Latent Diffusion [37] in multiple squared resolution settings (small, medium and large sizes according to their default training sizes). We evaluate them with the metric of Fréchet Inception Distance (FID) [13], which measures image quality and diversity. As shown in Table 1, our scaling factor universally improves FID for both Stable Diffusion and Latent Diffusion in all resolution settings, especially for the images with small resolution (a significant improvement from 74.6 to 41.9 for Stable Diffusion on the resolution 224 × 224 ). It is worth noting that the proposed method needs no additional training and achieves these in a plug-and-play manner with trivial complexity.

Besides FID, we also evaluate the degree of semantic alignment between text prompts and the corresponding synthesized images with CLIP scores [35], which utilizes contrastive learning to establish connections between visual concepts and natural language concepts. As shown in Table 2, our scaling factor achieves better CLIP scores than the original one in all resolution settings. It shows that our scaling factor performs better in the alignment of text prompts and the synthesized images.

Additionally, we present the results of user study in Table 3. We conduct a text-based pairwise preference test. That is, for a given sentence, we pair the image from our model with the synthesized image from the baseline model and ask 45 annotators in total to give each synthesized image a rating score for consistency. For each human annotators, we pay $15 for effective completeness of user study evaluation. We observe that users rate higher for our method, which suggests that the synthesized images from our method are more contextually coherent with texts than the baseline. Besides, with the refinement from the proposed scaling factor, the generated contents from our model are able to convey more natural and informative objects.

Ours

Stable

Diffusion

An octopus in the sea

A squirrel

eating a bug

A playground

with a dog

People on the

sunset beach

Street with cars

in daylight

Sunglasses in Brown

Kids doing homeworks

Rose for the graves

Wolverine Men's Chisel

Low Steel Toe Work Shoes

Bright green bedroom

Smiling duck

Silverstone Porsche

Auction

Ours

Stable

Diffusion

Ours

Stable

Diffusion

Train on the railway

A girl smiling

A cup of milk

on the table

Cats lying

on the ground

Wedding in church

Angry panda in the zoo

Qualitative comparison. In Figures 3 and 4, we show some visualization results for the qualitative comparison between two scaling factors with Stable Diffusion. Paired images are synthesized with the same seeds and text prompts. Please zoom in for the best visual effect. For Figure 3, we demonstrate results under the proposed and the original scaling factors for Stable Diffusion with 224 × 224 resolution, which is lower than the default training resolution 512 × 512 of Stable Diffusion. We observe that in the original Stable Diffusion setting, objects are either missed out (e.g., absent dog, people and octopus) or roughly depicted (e.g., crudely rendered squirrel, cars and person with sunglasses). Our interpretation is that for lower resolutions, according to Eq. (5), the model is synthesizing images with a less entropy during the inference phase when compared to its training phase, which implies that the model is suffering from deprived information. In this way, the model has more difficulty in generating complex objects and cannot generate attractive images. As for the results generated by the proposed scaling factor, we show that images are synthesized with better alignments with prompts (e.g., clear presence of dog, people and octopus) and objects are generated more vividly (e.g., realistic squirrel, cars and person with sunglasses). Note that our scaling factor also performs better in the illumination and shading because of the utilization of more aggregated

Ours

Stable

Diffusion

BECOMG

MORIG

TMORA

TTALP

STIY

BIABLG

thannaa

than thn

BIBLL

SRIYY

BCCOMNMORDATBCOUMIEY

theFAHAARIYTHAAIITINBEBLS

6.0

7.0

8.0

9.0

10.0

3.0

4.0

5.0

6.0

7.0

logN

Entropy (bits)

5.0

5.5

6.0

6.5

7.0

7.5

8.0

2.0

3.0

4.0

5.0

6.0

logN

Entropy (bits)

and contextualized information with higher entropy. Considering its flexibility, our method might offer valuable insights into the fields of inpainting [28, 56] and compositional scene generation [59].

For Figure 4, we demonstrate the results of Stable Diffusion under the proposed and the original scaling factors with the 768 × 768 resolution, which is higher than the default training resolution 512 × 512 of Stable Diffusion. Different from the rough objects in Figure 3, we note that objects are depicted with overwhelming details in a messy and disordered manner (e.g., repeated hands, a extended vehicle and messy seals). The reason is that for higher resolutions, according to Eq.(5), the model are bearing more entropy than it does in the training period. As a result, tokens are attending to numerous but redundant global information and the model depicts objects with burdensome details, leading to the patterns observed. For the results by our scaling factor, we note that the repeated patterns are greatly alleviated (e.g., a pair of normal hands and a regular vehicle) and synthesized objects are spatially arranged in order without losing the details (three seals well depicted in order).

4.2 Analysis

In this section, we conduct a comprehensive analysis of the practical implications of our theoretical findings. This analysis is of great importance due to the nature of our fundamental theoretical discovery outlined in Eq.(5) . As an asymptotic equality, it might exhibit limitations in practical scenarios where the token quantity, denoted as N , falls below a certain threshold. Thus, it becomes imperative to assess the feasibility and the applicability of our theoretical finding under such circumstances. Subsequently, we delve into an in-depth examination of the pivotal role played by the scaling factor

0

5

10

15

20

4.00

5.00

6.00

7.00

8.00

9.00

Layers_ID

Entropy (bits)

Resolution: 768 x 768

DM

Ours

0

5

10

15

20

0.00

2.00

4.00

6.00

Layers_ID

Entropy (bits)

Resolution: 224 x 224

in mitigating the fluctuations within the attention entropy, as presented in Section 3.2. This rigorous analysis serves to substantiate and validate the insights elucidated in the aforementioned section. To support our claims, we present visually intuitive representations of the empirical results, which unequivocally demonstrate the alignment between our statements and the experimental outcomes.

The relationship between the attention entropy and the token number. We conduct an experiment on text-to-image tasks to investigate the relation between the entropy and the token number N , which is statistically connected in Eq.(5). In particular, we synthesize images with various resolutions (including 224 2 , 384 2 , 448 2 , 512 2 and 768 2 for Stable Diffusion and 128 2 , 224 2 , 256 2 and 384 2 for Latent Diffusion model) and calculate their attention entropy during the inference period. We average the results across attention heads and layers for 5K samples on each resolution. Figure 5 illustrates that the average attention entropy has a linear relationship with the logarithm of the quantity of tokens for both diffusion models, which accords with their asymptotic relationship proposed in Eq.(5). The visualized results provide the empirical evidence that substantiates the efficacy of our theoretical findings and the feasible extensions for those resolutions close to the mentioned resolutions.

The role of scaling factor for stabilizing the attention entropy. We conduct an experiment on the mitigating effect of the proposed scaling factor onto the fluctuated attention entropy. Specifically, with two scaling factors, we synthesize images with a lower and a higher resolution (including 224 2 and 768 2 for Stable Diffusion) and record their attention entropy for each attention layers. We average the results across attention heads for 5K samples. As demonstrated in the Figure 6, the proposed scaling factor does contribute to an increase in the synthesis process for the resolution 224 2 and a decrease in that for the resolution 768 2 , which accords with our motivation in Section 3.2 and shows the efficacy of our scaling factor in mitigating the fluctuation of the attention entropy.

4.3 Difference with other candidate methods

In this section, we discuss upon the difference between the proposed method and other candidate methods for variable-sized image synthesis, including up/down-sampling methods and super-resolution methods. While all the mentioned methods share the same goal to synthesis images of variable sizes, there are three distinctive features setting our method apart from the rest.

Different aspect ratio. Other candidate methods do not support diffusion models to generate new images with a different aspect ratio. In comparison, our method could improve image synthesis in different aspect ratio, validated by both qualitative and quantitative experiments in Section 4.1.

Richness of visual information (important for higher resolutions). When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Under this circumstance, our method augments semantic information by enabling models to deal with more tokens in an adaptive manner, which enables images to possess a more extensive range of content. As shown in Figure 7, our method generates images with more

Ours

1024

⨉

1024

Up-sampled Stable Diffusion

1024

⨉

1024

L'adoption internationale.

Everyday Mango Poster

on the wall.

Out of the Fog.

Stable Diffusion

1024

⨉

1024

Stable Diffusion

512

⨉

512

EEVEYIAOAY

EVEVRYRAY

FFNEJM.ANO

EVEVRYRAY

FFNEJM-ANO

richness and expressiveness in semantics. In contrast, super-resolution and other methods scale up the original images and focus on better image clarity and finer details instead of richer semantics, introducing visual information at a different granularity level compared with the proposed method.

Time cost and memory usage (important for lower resolutions). For diffusion models adapted by our method, their time cost and spatial usage become proportional to the generation resolution, while down-sampling methods are constantly constrained to the fixed cost brought by training resolutions. As a result, our method could efficiently enable low resolution image synthesis especially on portable devices, which has a high demand for both time cost and memory usage other than image fidelity. For more quantitative results to support this, please refer to the Supplementary Materials.

5 Conclusion

In this work, we delves into a new perspective of adapting diffusion models to effectively synthesize images of varying sizes with superior visual quality, based on the concept of entropy. We establish a statistical connection between the number of tokens and the entropy of attention maps. Utilizing this connection, we give interpretations to the defective patterns we observe and propose a novel scaling factor for visual attention layers to better handle sequences of varying lengths. The experimental results demonstrate that our scaling factor effectively enhances both quantitative and qualitative scores for text-to-image of varying sizes in a training-free manner, while stabilizing the entropy of attention maps for variable token numbers. Moreover, we provide extensive analysis and discussion for the efficiency and the significance of our method.

Limitations One limitation of this study pertains to the metrics used for quantitative evaluation. While these metrics primarily evaluate the quality of the generated content, there is a lack of methodology for evaluating the fidelity of images for different resolutions. The same issue applies for the evaluation of repetitively disordered pattern, which is in part compensated by qualitative scores.

Acknowledgment

This work was supported in part by the National Key R&D Program of China (No.2021ZD0112803), the National Natural Science Foundation of China (No.62176060, No.62176061), STCSM project (No.22511105000), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0103), UniDT's Cognitive Computing and Few Shot Learning Project, and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.

References

Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483 , 2020.

Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. Entropy-based attention regularization frees unintended bias mitigation from lists. arXiv preprint arXiv:2203.09192 , 2022.

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014.

Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 , 2020.

Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI , pages 170-188. Springer, 2022.

Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8628-8638, 2021.

Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 , 2019.

Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusionbased semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427 , 2022.

Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2112.00029 , 2021.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Hamidreza Ghader and Christof Monz. What does attention in neural machine translation pay attention to? arXiv preprint arXiv:1710.03348 , 2017.

Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626 , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res. , 23(47):1-33, 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv preprint arXiv:2204.03458 , 2022.

Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: A magnification-arbitrary network for super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1575-1584, 2019.

Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In Proceedings of the IEEE international conference on computer vision , pages 1501-1510, 2017.

DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Blockrecurrent transformers. arXiv preprint arXiv:2203.07852 , 2022.

Zhiyu Jin, Xuli Shen, Bin Li, and Xiangyang Xue. Style spectroscope: Improve interpretability and controllability through fourier analysis. arXiv preprint arXiv:2208.06140 , 2022.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00364 , 2022.

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 , 2020.

Krzysztof Lis, Matthias Rottmann, Sina Honari, Pascal Fua, and Mathieu Salzmann. Attentropy: Segmenting unknown objects in complex scenes using the spatial attention entropy of semantic segmentation transformers. arXiv preprint arXiv:2212.14397 , 2022.

Peter J Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. arXiv preprint arXiv:1801.10198 , 2018.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Ming Lu, Hao Zhao, Anbang Yao, Yurong Chen, Feng Xu, and Li Zhang. A closed-form solution to universal style transfer. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5952-5961, 2019.

Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11461-11471, 2022.

Ye Ma, Jin Ma, Min Zhou, Quan Chen, Tiezheng Ge, Yuning Jiang, and Tong Lin. Boosting image outpainting with semantic layout prediction. arXiv preprint arXiv:2110.09267 , 2021.

Lovish Madaan, Srinadh Bhojanapalli, Himanshu Jain, and Prateek Jain. Treeformer: Dense gradient trees for efficient attention computation. arXiv preprint arXiv:2208.09015 , 2022.

Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations , 2021.

Byung-Doh Oh and William Schuler. Entropy-and distance-based predictors from gpt-2 attention patterns predict reading times over and above gpt-2 surprisal. arXiv preprint arXiv:2212.11185 , 2022.

Adam Pardyl, Grzegorz Rype´ s´ c, Grzegorz Kurzejamski, Bartosz Zieli´ nski, and Tomasz Trzci´ nski. Active visual exploration based on attention-map entropy. arXiv preprint arXiv:2303.06457 , 2023.

Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference on machine learning , pages 4055-4064. PMLR, 2018.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-based sparse attention with routing transformers. Transactions of the Association for Computational Linguistics , 9:53-68, 2021.

Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACMSIGGRAPH 2022 Conference Proceedings , pages 1-10, 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems , 35:36479-36494, 2022.

Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal , 27(3):379-423, 1948.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Jianlin Su. Revisiting attention scale operation from the invariance of entropy, Dec 2021. [Blog post]. Retrieved from https://kexue.fm/archives/8823 .

Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse sinkhorn attention. In International Conference on Machine Learning , pages 9438-9447. PMLR, 2020.

Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1052110530, 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems , 33:21665-21674, 2020.

Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1905-1914, 2021.

Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops , pages 0-0, 2018.

Yuxin Wang, Chu-Tak Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. What dense graph do you need for self-attention? In International Conference on Machine Learning , pages 22752-22768. PMLR, 2022.

Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034 , 2022.

Wenhan Xiong, Barlas O˘ guz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, and Yashar Mehdad. Simple local attentions remain competitive for long-context tasks. arXiv preprint arXiv:2112.07210 , 2021.

Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10561-10570, 2019.

Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023.

Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems , 33:17283-17297, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes both an analysis and a contribution to fix a problem found during the analysis.\n\nThey start with the premise that diffusion models should be able to generate arbitrary size images, and training specialized models for each image size is too expensive, which is correct. Using diffusion models trained for square images would be much cheaper and easier.\nThey identify two key problems when using square models to generate arbitrary aspect ratio images: incomplete/inadequate objects and repetitive/disorganized patterns.\nIn order to improve performance all around (quality, prompt following, etc.) they study entropy in the generated image.\nSpecifically, they note that as entropy rises tokens attend to wider regions, and relate this phenomenon to the problems delineated above.\nFinally, they find that simply proposing a scaling factor mitigates many of these issues.\n\nSTRENGTHS:\n1. The idea to add a scaling factor that counteracts entropy fluctuation in a training-free manner is really nice and the formulation is a principled approximation that is easily computable.\n2. The implementation is incredibly simple, a large plus.\n3. The work presents strong quantitative evaluations as well as a strong user evaluation that is commendable.\n4. The work presents a wide amount of qualitative samples that show improvement in very common problems for diffusion models such as double heads, double hands and other issues. Samples look good.\n\nWEAKNESSES:\n1. Hard to find many weaknesses, it's a strong paper that is well written, with a clear argument, clear analysis, clear solution that simply seems to work after both quantitative and user study evaluation (and with substantial qualitative samples showing edge cases).\n\nQUESTIONS:\n1. I'm assuming time complexity does not vary with the new scaling factor?\n\nLIMITATIONS:\n1. Good limitation section. Metrics are a bit of an issue but there is a strong user eval in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the issues of using a fixed-resolution diffusion model to generate varied-size images and proposes a scaling factor to stable the attention entropy which remedies the issue. The method is evaluated on text-to-image models when the inference resolution is moderately different than the model resolution and the results show the effectiveness of the proposed method compairing to directly using fixed-resolution diffusion model for varied-size synthesis.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow. The motivation is clearly explained and it is with some theoretical analysis.\n2. The proposed method is simple to implement, which does not require costly large scale training and could help varied-size synthesis with a fixed-resolution diffusion model.\n3. The effectiveness of changing the scale factor in attention for varied-size synthesis is an interesting observation.\n\nWEAKNESSES:\n1. Synthesizing images at different resolutions is an important research problem, but it is not very clear that why a fixed-resoltion pre-trained model should be used for this task (which is the main baseline in this work). To achieve this goal, there are many other candidate methods: (1) using diffusion-based (e.g. cascaded diffusion models) or GAN-based super-resolution method to upsample the output to some resolution of powers of 2, then downsample it to the query resolution. (2) using a any-scale super-resolution network (e.g. LIIF) to upsample the network result. It is not clear that why modifying fixed-resolution model is an important research topic if it's not shown to outperform other candidate methods.\n2. The test resolutions are still in a small changing range compared to the model resolution (about x0.5\\~x1.5), and the FID on x0.5 scale suggests that the quality could be significantly decreased when changing the scale to be different than training scale. Does this suggest that the proposed method does not work for a wider range of scales (for example, any-scale SR works typically evaluates x1\\~x4, or even extrapolates to x20\\~30)?\n\nQUESTIONS:\nSee weaknesses. It is not clear to me that, if a diffusion model only see faces at resolution 256 during training, how can its UNet synthesize a face at resolution 2048 with its kernels (trained for resolution 256) without repetitive patterns? Does the proposed method address this issue?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work adapts a pre-trained Stable Diffusion model for variable-resolution image generation. Since Stable Diffusion is trained on a fixed image resolution, naively varying the output size results in abnormal patterns in the images. This paper tracks the problem down to self-attention weights in the denoiser network, and identifies the so-called attention entropy as the root cause by proving that attention entropy is fundamentally correlated with image resolution. Drawing on this insight, the paper introduces resolution-dependent scaling to self-attentions to calibrate attention entropy throughout the generative process. The effectiveness of the proposed method is validated through extensive experiments.\n\nSTRENGTHS:\n- The method is training-free. It works by modifying the generative process of a pre-trained diffusion model at inference time.\n\n- The method reveals a strong connection between self-attention entropy and image quality. It builds on the key finding that attention entropy is correlated with image resolution, and artifacts in low / high-resolution images can be attributed to mis-calibrated attention entropy. It thus attempts to calibrate attention entropy across various image resolutions using a resolution-dependent scaling factor. Turning theoretical insights into a simple, actionable solution is a key strength of the paper.\n\n- The effectiveness of the method is validated by qualitative, quantitative and user study results. The improvement over the Stable Diffusion baseline seems quite consistent.\n\nWEAKNESSES:\n- A few simple baselines are missing. For example, one can simply down-sample a 512x512 image to reach a lower resolution. Similarly, one can generate a 512x512 image and subsequently up-sample it using an off-the-shelf super-resolution model. In fact, Stable Diffusion supports super-resolution and uneven aspect ratios with community effort (check out AUTOMATIC1111). To justify the main contribution of the paper (i.e., improved generation quality of variable-sized images), it is important to show that the proposed method performs equally well, or even better, compared to these simple baselines.\n\n- Arguably, generating high-resolution images is of greater interest to the community. To this end, it would be interesting to probe the limit of the proposed method. That is, what is the highest resolution it can handle without introducing noticeable artifacts? Most high-resolution images in the paper are 768x768. There is one image in the supplement at the resolution of 1024x1024. The proposed method will generate more impact if it can further grow the image resolution.\n\n- According to Figure 5 (left), attention entropy is up by 1 bit as image resolution grows from 512 to 768. However, Figure 6 (right) seems to suggest that applying the scaling factor only reduces entropy by a very small amount. My early impression is that the scaling factor aims to restore entropy to the level of 512x512 images. I am curious about what causes this discrepancy.\n\nQUESTIONS:\nPlease see the section above for questions.\n\nLIMITATIONS:\nThe paper discussed the limitations and societal impact of the method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new scaling factor for attention based text-to-image generative models in order to handle variable sized generations. The authors establish the relationship between attention entropy and token size and use this newly found relationship to design a scaling factor that takes into account the image resolution. The authors also empirically show the effectiveness of this scaling factor by comparing FIDs, CLIP scores and qualitative examples.\n\nSTRENGTHS:\nThe proposed scaling factor is very simple and easy to implement. It does not require model architecture modifications or any training of the pretrained model. The qualitative results also look very promising. Therefore I think this can be easily applied in many existing models. Overall the paper is well organized and well written.\n\nWEAKNESSES:\n1. The main formula (Equation 6 and 7) is not very well explained. There are a lot of conjectures and approximations without justifications.\n2. The authors did not compare with synthesis in fixed resolution and then performing super-resolution/downsampling. Since the resolution experimented in this paper is not very far away from the pretrained resolution, it is highly likely that super-resolution/downsampling may work just as well.\n3. The authors should also compare their method with other machine learning based methods such as MultiDiffusion (https://multidiffusion.github.io/)\n4. The compensation for human annotator is not mentioned\n5. Typo in Equation 1,3: j is used in both the numerator and the denominator of Aij\n\nQUESTIONS:\n1. Related to Weakness (1), Section 3.2 is still a little bit confusing to me, can the authors explain more on the connection between Equation 5, 6 and 7? I.e. How did the authors come up with Equation 6 and 7?\n2. Related to Weakness (2), can the authors compare their method with synthesis in fixed pretrained resolution and perform super-resolution/downsampling? Can authors also compare to more dramatic resolution changes (eg. 512 => 128 or 512 => 1024)?\n3. Related to Weakness (3), can the authors compare their method with MultiDiffusion?\n4. Is FID appropriate for evaluating wrt different resolutions? Since it is trained on fixed resolution images. And how did the authors select the reference/ground truth images for FID evaluation? Did they uniformly rescale to the desired resolution or did they subsampled based on the native resolution? It may have a difference in the evaluation results.\n5. Although CLIP score is a reasonable indicator, I don’t think it is a perfect metric especially when object count is involved. Have the authors considered using off-the-shelf object detectors as one of the methods to quantitatively measure the performance?\n\nLIMITATIONS:\nRelated to Weakness (4), the compensation for human annotators is not specified in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}