SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, we present the results of user study in Table 3. We conduct a text-based pairwise preference test. That is, for a given sentence, we pair the image from our model with the synthesized image from the baseline model and ask 45 annotators in total to give each synthesized image a rating score for consistency. For each human annotators, we pay $15 for effective completeness of user study evaluation. We observe that users rate higher for our method, which suggests that the synthesized images from our method are more contextually coherent with texts than the baseline. Besides, with the refinement from the proposed scaling factor, the generated contents from our model are able to convey more natural and informative objects.
Figure 3: Qualitative comparison on the scale factors for the resolution  $224 \times 224$ . The original scaling factor misses out content or roughly depicts the objects in prompts while our scaling factor manages to synthesize visual concepts in high fidelity and better illumination. Please zoom in for better visual effect.
Qualitative comparison. In Figures 3 and 4, we show some visualization results for the qualitative comparison between two scaling factors with Stable Diffusion. Paired images are synthesized with the same seeds and text prompts. Please zoom in for the best visual effect. For Figure 3, we demonstrate results under the proposed and the original scaling factors for Stable Diffusion with  $224 \times 224$  resolution, which is lower than the default training resolution  $512 \times 512$  of Stable Diffusion. We observe that in the original Stable Diffusion setting, objects are either missed out (e.g., absent dog, people and octopus) or roughly depicted (e.g., crudely rendered squirrel, cars and person with sunglasses). Our interpretation is that for lower resolutions, according to Eq. (5), the model is synthesizing images with a less entropy during the inference phase when compared to its training phase, which implies that the model is suffering from deprived information. In this way, the model has more difficulty in generating complex objects and cannot generate attractive images. As for the results generated by the proposed scaling factor, we show that images are synthesized with better alignments with prompts (e.g., clear presence of dog, people and octopus) and objects are generated more vividly (e.g., realistic squirrel, cars and person with sunglasses). Note that our scaling factor also performs better in the illumination and shading because of the utilization of more aggregated
Ours
BCCOMN MORD AT BCOUMEY (B) GISHA A DIX TELA A TIRITAT ABIL BIL
Figure 4: Qualitative comparison on the scale factors for the resolution  $768 \times 768$ . The original scaling factor generates objects in repeated and messy patterns while our scaling factor manages to spatially arrange the visual concepts in clear order. Please zoom in for better visual effect.
logN
Figure 5: Analysis results on the average attention entropy with respect to the number of tokens for Stable Diffusion model (left) and latent diffusion model (right).  $N$  refers to the number of tokens. As shown in the figure, average attention entropy has a linear relationship with the logarithm of  $N$ , which confirms the effectiveness of our asymptotic equality in Eq.(5)
logN
and contextualized information with higher entropy. Considering its flexibility, our method might offer valuable insights into the fields of inpainting [28, 56] and compositional scene generation [59].
For Figure 4, we demonstrate the results of Stable Diffusion under the proposed and the original scaling factors with the  $768 \times 768$  resolution, which is higher than the default training resolution  $512 \times 512$  of Stable Diffusion. Different from the rough objects in Figure 3, we note that objects are depicted with overwhelming details in a messy and disordered manner (e.g., repeated hands, a extended vehicle and messy seals). The reason is that for higher resolutions, according to Eq.(5), the model are bearing more entropy than it does in the training period. As a result, tokens are attending to numerous but redundant global information and the model depicts objects with burdensome details, leading to the patterns observed. For the results by our scaling factor, we note that the repeated patterns are greatly alleviated (e.g., a pair of normal hands and a regular vehicle) and synthesized objects are spatially arranged in order without losing the details (three seals well depicted in order).
In this section, we conduct a comprehensive analysis of the practical implications of our theoretical findings. This analysis is of great importance due to the nature of our fundamental theoretical discovery outlined in Eq.(5). As an asymptotic equality, it might exhibit limitations in practical scenarios where the token quantity, denoted as  $N$ , falls below a certain threshold. Thus, it becomes imperative to assess the feasibility and the applicability of our theoretical finding under such circumstances. Subsequently, we delve into an in-depth examination of the pivotal role played by the scaling factor
Figure 6: Analysis results on the average attention entropy with respect to two scale factors for two resolution. Layer ID refers to the index number of self-attention layers in the model and DM stands for original diffusion model. The figure illustrates that the proposed scaling factor mitigates the fluctuation of attention entropy.
in mitigating the fluctuations within the attention entropy, as presented in Section 3.2. This rigorous analysis serves to substantiate and validate the insights elucidated in the aforementioned section. To support our claims, we present visually intuitive representations of the empirical results, which unequivocally demonstrate the alignment between our statements and the experimental outcomes.
The relationship between the attention entropy and the token number. We conduct an experiment on text-to-image tasks to investigate the relation between the entropy and the token number  $N$ , which is statistically connected in Eq.(5). In particular, we synthesize images with various resolutions (including  $224^2$ ,  $384^2$ ,  $448^2$ ,  $512^2$  and  $768^2$  for Stable Diffusion and  $128^2$ ,  $224^2$ ,  $256^2$  and  $384^2$  for Latent Diffusion model) and calculate their attention entropy during the inference period. We average the results across attention heads and layers for 5K samples on each resolution. Figure 5 illustrates that the average attention entropy has a linear relationship with the logarithm of the quantity of tokens for both diffusion models, which accords with their asymptotic relationship proposed in Eq.(5). The visualized results provide the empirical evidence that substantiates the efficacy of our theoretical findings and the feasible extensions for those resolutions close to the mentioned resolutions.
The role of scaling factor for stabilizing the attention entropy. We conduct an experiment on the mitigating effect of the proposed scaling factor onto the fluctuated attention entropy. Specifically, with two scaling factors, we synthesize images with a lower and a higher resolution (including  $224^2$  and  $768^2$  for Stable Diffusion) and record their attention entropy for each attention layers. We average the results across attention heads for 5K samples. As demonstrated in the Figure 6, the proposed scaling factor does contribute to an increase in the synthesis process for the resolution  $224^2$  and a decrease in that for the resolution  $768^2$ , which accords with our motivation in Section 3.2 and shows the efficacy of our scaling factor in mitigating the fluctuation of the attention entropy.
4.3 Difference with other candidate methods
In this section, we discuss upon the difference between the proposed method and other candidate methods for variable-sized image synthesis, including up/down-sampling methods and super-resolution methods. While all the mentioned methods share the same goal to synthesis images of variable sizes, there are three distinctive features setting our method apart from the rest.
Different aspect ratio. Other candidate methods do not support diffusion models to generate new images with a different aspect ratio. In comparison, our method could improve image synthesis in different aspect ratio, validated by both qualitative and quantitative experiments in Section 4.1.
Richness of visual information (important for higher resolutions). When users generate images with higher resolutions, what they are expecting is not only more pixels but also richer semantic information in images. Under this circumstance, our method augments semantic information by enabling models to deal with more tokens in an adaptive manner, which enables images to possess a more extensive range of content. As shown in Figure 7, our method generates images with more
Stable Diffusion  $512\times 512$
L'adoption internationale.
Everyday Mango Poster on the wall.
Out of the Fog.
Up-sampled Stable Diffusion  $1024\times 1024$
Figure 7: Qualitative results to illustrate the difference in visual information richness between up-sampled methods, the original Stable Diffusion and the proposed method. Note that our method (right) introduces more visual information with a different level of granularity compared with up-sampling and super-resolution methods (left). Additionally, our method could better deal with repetitively disordered pattern emerging in the original Stable Diffusion model (middle).
Stable Diffusion  $1024\times 1024$
Ours  $1024\times 1024$
richness and expressiveness in semantics. In contrast, super-resolution and other methods scale up the original images and focus on better image clarity and finer details instead of richer semantics, introducing visual information at a different granularity level compared with the proposed method.
Time cost and memory usage (important for lower resolutions). For diffusion models adapted by our method, their time cost and spatial usage become proportional to the generation resolution, while down-sampling methods are constantly constrained to the fixed cost brought by training resolutions. As a result, our method could efficiently enable low resolution image synthesis especially on portable devices, which has a high demand for both time cost and memory usage other than image fidelity. For more quantitative results to support this, please refer to the Supplementary Materials.
In this work, we delve into a new perspective of adapting diffusion models to effectively synthesize images of varying sizes with superior visual quality, based on the concept of entropy. We establish a statistical connection between the number of tokens and the entropy of attention maps. Utilizing this connection, we give interpretations to the defective patterns we observe and propose a novel scaling factor for visual attention layers to better handle sequences of varying lengths. The experimental results demonstrate that our scaling factor effectively enhances both quantitative and qualitative scores for text-to-image of varying sizes in a training-free manner, while stabilizing the entropy of attention maps for variable token numbers. Moreover, we provide extensive analysis and discussion for the efficiency and the significance of our method.
Limitations One limitation of this study pertains to the metrics used for quantitative evaluation. While these metrics primarily evaluate the quality of the generated content, there is a lack of methodology for evaluating the fidelity of images for different resolutions. The same issue applies for the evaluation of repetitively disordered pattern, which is in part compensated by qualitative scores.
This work was supported in part by the National Key R&D Program of China (No.2021ZD0112803), the National Natural Science Foundation of China (No.62176060, No.62176061), STCSM project (No.22511105000), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0103), UniDT's Cognitive Computing and Few Shot Learning Project, and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
[1] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483, 2020.
[2] Giuseppe Attanasio, Debora Nozza, Dirk Hovy, and Elena Baralis. Entropy-based attention regularization frees unintended bias mitigation from lists. arXiv preprint arXiv:2203.09192, 2022.
[3] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
[4] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
[5] Lucy Chai, Michael Gharbi, Eli Shechtman, Phillip Isola, and Richard Zhang. Any-resolution training for high-resolution image synthesis. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVI, pages 170-188. Springer, 2022.
[6] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8628-8638, 2021.
[7] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019.
[8] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion-based semantic image editing with mask guidance. arXiv preprint arXiv:2210.11427, 2022.
[9] Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated butterfly: Simple and efficient sparse training for neural network models. arXiv preprint arXiv:2112.00029, 2021.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[11] Hamidreza Ghader and Christof Monz. What does attention in neural machine translation pay attention to? arXiv preprint arXiv:1710.03348, 2017.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv:2208.01626, 2022.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Jonathan Ho, William Chan, Chitwan Sahara, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[50] Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10521-10530, 2019.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[52] Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. Advances in Neural Information Processing Systems, 33:21665-21674, 2020.
[53] Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1905-1914, 2021.
[54] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018.
[55] Yuxin Wang, Chu-Tak Lee, Qipeng Guo, Zhangyue Yin, Yunhua Zhou, Xuanjing Huang, and Xipeng Qiu. What dense graph do you need for self-attention? In International Conference on Machine Learning, pages 22752-22768. PMLR, 2022.
[56] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. arXiv preprint arXiv:2212.05034, 2022.
[57] Wenhan Xiong, Barlas Oğuz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Wen-tau Yih, and Yashar Mehdad. Simple local attentions remain competitive for long-context tasks. arXiv preprint arXiv:2112.07210, 2021.
[58] Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10561-10570, 2019.
[59] Jinyang Yuan, Tonglin Chen, Bin Li, and Xiangyang Xue. Compositional scene representation learning via reconstruction: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[60] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-17297, 2020.