Title: How far can we go with ImageNet for Text-to-Image generation?
Venue: NeurIPS 2025
Accepted: no
Keywords: Text-to-image diffusion models
Abstract: Recent text-to-image (T2I) generation models have achieved remarkable results by training on billion-scale datasets, following a `bigger is better' paradigm that prioritizes data quantity over availability (closed vs open source) and reproducibility (data decay vs established collections). We challenge this established paradigm by demonstrating that one can match or outperform models trained on massive web-scraped collections, using only ImageNet enhanced with well-designed text and image augmentations. With this much simpler setup, we achieve a $+1\%$ overall score over SD-XL on GenEval and $+0.5\%$ on DPGBench while using just 1/10th the parameters and 1/1000th the training images. 
  This opens the way for more reproducible research as ImageNet is a widely available dataset and our standardized training setup does not require massive compute resources.

1

Abstract

1 Introduction 14

The prevailing wisdom in text-to-image (T2I) generation holds that larger training datasets inevitably 15 lead to better performance. This 'bigger is better' paradigm has driven the field to billion-scale 16 image-text paired datasets like LAION-5B [43], DataComp-12.8B [10] or ALIGN-6.6B [36]. While 17 this massive scale is often justified as necessary to capture the full text-image distribution, in this 18 work, we challenge this assumption and argue that data quantity overlooks fundamental questions of 19 data efficiency and quality in model training. 20

21

22

23

24

25

26

27

28

Our critique of the data-scaling paradigm comes from a critical observation: current training sets are either closed-source or rapidly decaying which makes results impossible to fully reproduce, let alone compare fairly. As such, the community of T2I generation is in dire need of a standardized training setup to foster open and reproducible research. Luckily, Computer Vision already has such dataset in ImageNet [41] that has been the gold standard in many tasks for many years. It is widely available and its strength and limitations are well known. Furthermore, it is heavily used in class-conditional image generation [35, 20], which makes its evaluation metrics more familiar. This begs the question of how far can we go with ImageNet for text-to-image generation?

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

Our findings are that we can indeed get a surprisingly competitive model by training solely on ImageNet. As shown on Figure 1, we can achieve excellent visual quality. Additionally we also achieve very competitive scores on common benchmarks such as GenEval [11] and DPGBench [18], matching or even surpassing popular models that are trained on much more data and at a far greater cost, such as SDXL [37] and Pixartα [3] (see Figure 2). However, this does not come without any hurdles. In this paper we analyze the challenges of training using ImageNet only and propose successful strategies to overcome them. Our strategies allow us to train models of smaller size (about 300M parameters) on a reasonable compute budget (about 500 H100 hours) making it accessible to more research teams, while not compromising on the capabilities.

Our contributions are thus the following:

We analyze the shortcomings of training T2I diffusion models on ImageNet and propose mitigation strategies.

Then, we propose a standardized training setup using only images from ImageNet, providing accessible and reproducible research for T2I generation.

We provide several models in the 300M-400M parameters range generating high quality images and outperforming competing models that are 10 times the size and trained on 1000 times more data.

To commit to open and reproducible science, all our training data are hosted at https:// 46 huggingface.co/datasets/anonymous_for_review and all our code and models are hosted at 47 https://github.com/anonymous_for_review . 48

In the next section, we outline the challenges in using ImageNet for T2I generation, and then evaluate 49 mitigation strategies for each of them. We then combine them in a complete training recipe that we 50 use to train several models of varying resolution. We compare them against the state of the art, with 51 excellent results. Finally, we discuss the related work before we conclude. 52

53

10

6

10

7

10

8

10

9

10

10

0

.

4

0

.

5

0

.

6

Ours

Pixart-

α

CAD

SDXL

SD2.1

Train Dataset Size (log. scale)

Geneval Overall scores

10

6

10

7

10

8

10

9

10

10

0

.

6

0

.

7

0

.

8

Ours

Pixart-

α

SDXL

SD 1.5

Train Dataset Size (log. scale)

54

2 Adopting ImageNet for Text-to-Image generation

We focus on training text-to-image models using ImageNet, a small, open-source and widely accepted 55 data collection. We first discuss the evaluation criterions and then gradually pinpoint the major 56 limitations in setting up a T2I diffusion model using ImageNet. To overcome these limitations, we 57 show that well-crafted augmentations can bring forth a compositionally accurate T2I model with data 58 constraints. For our analysis, we leverage two architectures: (1) DiT-I (our adaptation of DiT [35] to 59 handle text) and (2) CAD-I [6]. The suffix "I" is added to indicate the model is being trained only on 60 ImageNet. 61

62

63

64

65

66

Evaluation: Image-quality . We specifically assess the generation quality of both in-distribution (w.r.t Imagenet-50k validation set) and zero-shot (MSCOCO-30k captions validation set [30]). Specifically, we adopt: (1) FID [16] using both standard Inception-v3 and Dino-v2 backbones, (2) Precision [27], (3) Recall [27], (4) Density [34], and (5) Coverage [34]. These are all calculated with Dino-v2 features.

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

Evaluation: Compositionality . To understand the text-image alignment capabilities and image composition prowess, we adopt (1) CLIPScore [15] and (2) Jina-CLIP Score [26] on both MSCOCO30k and ImageNet validation set, (3) GenEval [11] and (4) DPGBench [18].

Evaluation: Aesthetic-quality to assess the human aesthetics of generated images. In line with prior works [9], to assess the aesthetical understanding of the generated images we also adopt: (1) PickScore [25], (2) Aesthetics Score [43], and (3) VILA Score [22] using PartiPrompts [53].

Based on these evaluation strategies that assess both image quality and compositional accuracy, we can systematically identify the key limitations of training T2I diffusion models on ImageNet. Our analysis reveals several ImageNet-specific challenges that must be addressed to achieve high-performance generation with limited data.

2.1 Text challenges

Challenge: Absence of captions. Class-conditional models trained with ImageNet have shown exceptional generation capabilities [35, 33] However, extending this use to T2I generation is difficult since ImageNet, being a classification dataset, lacks any sort of caption corresponding to its images. To adopt ImageNet for T2I generation, similar to prior works [38], one could build captions by a very simple strategy of 'An image of <class name>' (denoted AIO). However, this results in very poor generation capabilities as shown in Table 1. This can be mainly attributed to the two major shortcomings of AIO captions for ImageNet: First, AIO captions lack vocabulary. They contain only roughly a thousand words corresponding to the concepts of the classes and thus lack attributes, spatial relations, etc. This constraint on the diversity in the text-condition space leads to a clear bottleneck in text understanding. Second, there is often more content in the image than just the class. For example, a caption 'an image of golden retriever ' mentions the class name but leaves out details and concepts that could be in the background. This lack of details leads to spurious correlation where the model can learn to associate unrelated visual pattern (e.g., grass texture) to the class name (e.g., golden retriever ) because the text for this concept is never mentioned in the text space. Finally, despite the presence of humans in the images, ImageNet does not contain a ' person ' class, resulting in humans not being represented in the AIO text space. This issue extends to many categories ( road , water , etc), as ImageNet is an object-centric dataset.

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

Solution: Long informative captions . To overcome this challenge, we employ a synthetic captioner [32] (TA for Text Augmentation ) to generate comprehensive captions that capture: (i) Scene composition and spatial relationships ; (ii) Background elements and environmental context ; (iii) Secondary objects and participants ; (iv) Visual attributes (color, size, texture); and (v) Actions and interactions between elements.

We compare the gains attributed to long captions both quantitatively (Table 1) as well as qualitatively (Figure 4: row 1-2). For ImageNet-Val set, we observe that models trained with long captions significantly improves performance, resulting in lower FIDs of 6.29 for DiT-I and 6.16 for CAD-I in contrast to 20.14 for DiT-I and 85 for CAD-I on AIO captions. As a point of reference, we remind the reader that models of this size (below 0.5B parameters) typically have an FID of 9 using the class-conditional setup [35]. The aesthetic metrics (PickScore, Aesthetic Score, and VILA Score), offer additional insights, highlighting the superiority of TA models, which consistently outperform their AIO counterparts. For COCO test set - which is a zero-shot task for our training, this trend is all the more dramatic. The TA models are the only ones able to correctly follow the prompt as attested by the much improved Jina CLIP score (DiT-I from 22.42 to 38.45 ; CAD-I from 14.06 to 37.85 ), while keeping similar image quality.

Regarding text-image alignment and compositionality, models trained with longer captions benefit from the added information, evidenced by the improvement in GenEval overall score from (DiT-I from 0.11 to 0.55 ; CAD-I from 0.17 to 0.55 ) (see Table 1).

2.2 Image challenges

Using long, informative captions (TA) significantly enhances both generation quality and compositional alignment. But training text-to-image diffusion models on ImageNet only still faces two critical limitations: early overfitting and poor compositional generalization.

Limitation: Early overfitting . Models trained on ImageNet with long captions (TA) demonstrate promising initial performance. However, due to the relatively small scale of ImageNet (only 1.2 million images) they begin overfitting at approximately 200k training steps (see Figure 3).

Limitation: Restricted Complex Compositionality Abilities . ImageNet's object-centric nature presents a challenge for learning complex compositions. Even with enhanced textual descriptions via TA captioning, models still struggle with spatial relationships, attribute binding, and multi-object compositions. This limitation manifests in lower GenEval scores for compositional prompts involving multiple objects, color attribution, and positional relationships, as shown in Figure 4

Solution: Image Augmentation (IA) . To reduce overfitting and improve compositional reasoning, we investigate the use of image augmentations during training. We experiment with two augmentation strategies. Details about implementation and training pipeline are given in Appendices E and F.

CutMix [55]: For each image in the dataset, we randomly select an image from a different class and overlay a smaller version of it onto the original image. A caption is generated using the CutMix image as input. This technique introduces additional variability in the training data.

Crop : During training, we randomly mask a portion of the image tokens such that the model is exposed only to a local square crop of the original image. We add some crop coordinates tokens to the captions of the image. This augmentation encourages the model to decouple

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

We take a DiT-I checkpoint at 256 2 resolution, trained with TA + IA for 250,000 steps, and continue 165 training it at 512 2 resolution for an additional 100,000 steps on the same data. Both the pretraining and 166 the fine-tuning use the Crop image augmentation strategy for simplicity. This fine-tuning procedure 167 requires no changes to the text encoder or transformer backbone, aside from adjusting the image 168 tokenization to handle the larger input size. 169

object features from their background context, and to learn correspondences between partial visual elements and specific text tokens.

In Figure 3, we plot the evolution of FID and GenEval scores over training steps for the CAD-I architecture. Training with TA alone leads to early overfitting: we observe a sharp rise in FID after 200k steps. In contrast, models trained with TA+CutMix or TA+Crop maintain significantly lower FID curves for longer, with a delayed onset of overfitting.

Table 2 assesses the impact of image augmentation on GenEval metrics. Image augmentation (both CutMix and Crop) leads to a notable improvement in the GenEval overall score, with an increase of 2 points. Notably, the Two Objects sub-task sees a +7 point increase for both architectures, CAD-I sees a +9 point gain in Position, and DiT-I gains +6 points in Color Attribution.

These improvements in compositional metrics are achieved while maintaining or improving image quality, as measured by FID scores and Aesthetic metrics in Table 3. The qualitative examples in Figure 4 (rows 2-3) further demonstrate the enhanced compositional capabilities, with models trained using augmentation techniques producing more accurate representations of complex prompts. This improvement is particularly evident in the pirate ship scene: while the TA model generates a ship awkwardly positioned with a bowl of soup, the TA + IA model creates a more natural composition with the pirate ship appropriately sailing in the bowl. Similarly, the hedgehog and hourglass example shows more refined details and aesthetically pleasing composition with TA + IA, whereas the TA model struggles to render a recognizable hedgehog.

2.3 Scaling to higher resolution

All experiments discussed thus far were conducted at a resolution of 256 2 . We then explore whether high-resolution generation is feasible under the same data constraints. We investigate if this setup-utilizing only ImageNet with text and image augmentations-can scale to a 512 2 resolution without compromising performance or requiring additional supervision.

100

200

300

400

17

18

19

20

Training Steps in thousands of steps

FID Inc. ImageNet Val

TA

TA+IA

100

200

300

400

500

0

.

45

0

.

5

0

.

55

0

.

6

Training Steps in thousands of steps

GenEval score

TA

TA+IA

TA+IA

A ship sailing

on a soup

A hedgehog and

an hourglass

A teddy bear

driving a motorbike

A teapot and

some cookies

A Goat on a

mountain top

AIO

TA

173

Table 5. It exhibits slight but consistent gains on the GenEval benchmark over its

256

2

initialization

3 Comparison with SOTA 176

Qualitative results. On Figure 5, we show a comparison of images generated by our 512 DiT-I 178 model and SDXL. We are able to achieve similar visual quality as SDXL, with a tendency to generate 179 more photorealistic images. In contrast, SDXL is more likely to generate synthetic images (drawings, 180 painting, digital art). Given that ImageNet is overwhelmingly composed of photograph, this difference 181 of bias is likely to come from the dataset. In addition, we are slightly better at handling prompt 182 following (e.g., on the first image, only a single corgi, with the correct hat color ; lack of reflection on 183 the bird in the fountain ; lack of geyser on the last image). 184

Quantitative results: Comparison to the state of the art on GenEval and DPG benchmarks. 185 We test the composition ability of both our best DiT-I and CAD-I models, trained with TA + IA, 186 on the GenEval and DPGBench benchmarks and compare our performances to the ones of popular 187 state-of-the-art models. 188

GenEval. Table 5 reports the results on GenEval benchmark. We observe that our high-resolution 189 512 model ( 0.56 ) performs better on average than SD1.5 ( 0.43 ), Pixsartα ( 0.48 ), SD2.1 ( 0.50 ) and 190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

SDXL ( 0.55 ) in their native resolution. The striking improvements of our model are in the position attribute and color attribution where our model achieves more than +10 w.r.t SDXL. In lower 256 2 resolution the scores are even more distinctive as our models (CAD-I: 0.57 ; DiT-I: 0.58 ) outperform all the other popular models (SD1.5 ( 0.13 ), PixArtα ( 0.48 ), and CAD ( 0.50 )). In both 256 2 and 512 2 resolution our models reaches the performance of SDXL in its native resolution while having 10x fewer parameters and being trained on only 0.1% of the data .

DPGBench. Table 6 reports the results on DPGBench, a recent benchmark similar to Geneval but with a more complex prompt set. We observe similar trends as for GenEval: compared to the current leaderboard, we achieve an overall accuracy of 75.1% with DiT-I, which improves over SDXL by 0.5% . We reach an overall score of 79.94% for CAD-I, outperforming SDXL by +5% and PixArtα by +8% . Impressively, our models reach accuracies comparable to that of Janus [52], a 1.3B parameters VLM with generation capabilities. Notably, both our models are particularly good at relations, achieving state-of-the-art of 93.5% for CAD-I and 92.2% for DiT-I. Even at higher 512 2 resolution our model achieves better overall score ( 75.14% ) than SDXL ( 74.65% ), even with only 100 k steps of high-resolution fine-tuning.

4 Related Work

Diffusion Models. [46, 17, 45] have demonstrated remarkable success across various domains [19, 4, 7]. While image generation remains their most prominent application [5, 46, 21], text-to-image (T2I) synthesis [40, 42, 39] has emerged as a particularly impactful use case. These models operate by learning to reverse a gradual Gaussian noise corruption process. At extreme noise levels, the model effectively samples from a standard normal distribution to produce realistic images. The core optimization objective is:

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

where x t = √ γ ( t ) x 0 + √ 1 -γ ( t ) /epsilon1 denotes the noised image at timestep t , x 0 the original image, c the corresponding condition (such as text), /epsilon1 is standard normal noise, /epsilon1 θ the learned noise predictor, and γ ( t ) the variance schedule.

Computational Efficiency. Traditional diffusion models require substantial computational resources, with leading implementations consuming hundreds of thousands of GPU hours [40]. Recent advances have significantly improved training efficiency. [51, 54] identified limitations in the diffusion loss's representation learning capabilities, demonstrating that supplementary representation losses accelerate convergence. [3] achieved dramatic compute reduction by repurposing class-conditional models for text-to-image generation. [6] introduced architectural improvements and coherence-aware mechanisms, matching Stable Diffusion's performance [40] with 100x fewer GPU hours.

Data Efficiency. Early T2I models relied on billion-scale web-scraped datasets [40], creating accessibility barriers due to storage requirements and reproducibility challenges from copyright restrictions. [3] pioneered dataset reduction using 20M high-quality images from recaptioned SAM data [24], though portions remain proprietary. Subsequent work explored CC12M [2, 13, 6] and YFCC100M's public subset [47, 12], revealing overfitting below 10M samples. Our approach diverges by leveraging ImageNet [41] - a reproducible, well-established benchmark with standardized metrics [16]. We transform this classification dataset into T2I training data through synthetic captions and image augmentations.

Synthetic captions. Synthetic image captioning has benefited several tasks. For instance, visual question answering [44] and visual representation learning [48] achieve state-of-the-art performances by enhancing the captioning output of Vision-Language Models (VLMs) [28, 44]. Similarly, training with synthetic captions for text-to-image generation is becoming the defacto protocol for large diffusion models, such as DALL-E [1], Pixartα [3] and Stable Diffusion-3 [8]. More recently, some approaches [31, 29] extend this approach by training text-to-image (T2I) models on multi-level captions. Inspired by these, our method deploys the popular LLaVA captioner [32] to augment existing textual captions and use them to train text-to-image generation models.

5 Conclusion

In this work, we challenged the prevailing wisdom that billion-scale datasets are necessary for high-quality text-to-image generation. These large-scale datasets are usually either closed sourced or rapidly decaying, which threatens openness and reproducibility in text-to-image generation research. Instead, we show that it is possible to train smaller models to high quality using ImageNet only. We analyze the shortcomings of ImageNet for text-to-image generation and propose data-augmentation strategies to overcome these limitations.

We thus propose a standardize text-to-image training setup that leads to models capable of generating high quality images while being excellent at prompt following. This is attested by results on common benchmarks ( 56% or +1% on GenEval and 75 . 1% or +0 . 5% on DPGBench) on which we are able to outperform popular models such as SDXL.

The implications of our work extend beyond just computational efficiency, open science and reproducibility. By showing that smaller datasets can achieve state-of-the-art results, we open new possibilities for specialized domain adaptation where large-scale data collection is impractical. Our work also suggests a path toward more controllable and ethical development of text-to-image models, as smaller datasets enable more thorough content verification and bias mitigation.

Looking forward, we believe our results will encourage the community to reconsider the 'bigger is better' paradigm. Future work could explore additional augmentation strategies, investigate the theoretical foundations of data efficiency, and develop even more compact architectures optimized for smaller datasets. Ultimately, we hope this work starts a shift toward more sustainable and responsible development of text-to-image generation models.

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

References

James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. OpenAI , 2023.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixartalpha : Fast training of diffusion transformer for photorealistic text-to-image synthesis. ICLR , 2023.

Robin Courant, Nicolas Dufour, Xi Wang, Marc Christie, and Vicky Kalogeiton. Et the exceptional trajectories: Text-to-camera-trajectory generation with character awareness. In European Conference on Computer Vision , 2025.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS , 2021.

Nicolas Dufour, Victor Besnier, Vicky Kalogeiton, and David Picard. Don't drop your samples! coherence-aware training benefits conditional diffusion. In CVPR , 2024.

Nicolas Dufour, David Picard, Vicky Kalogeiton, and Loic Landrieu. Around the world in 80 timesteps: A generative approach to global visual geolocation. arXiv , 2024.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proc. ICML , 2024.

Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata. Reno: Enhancing one-step text-to-image models through reward-based noise optimization. Neural Information Processing Systems (NeurIPS) , 2024.

Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. NeurIPS , 2023.

Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems , 36, 2024.

Aaron Gokaslan, A Feder Cooper, Jasmine Collins, Landan Seguin, Austin Jacobson, Mihir Patel, Jonathan Frankle, Cory Stephenson, and Volodymyr Kuleshov. Commoncanvas: Open diffusion models trained on creative-commons images. In CVPR , 2024.

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Joshua M Susskind, and Navdeep Jaitly. Matryoshka diffusion models. In ICLR , 2023.

Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. Proc. EMNLP , 2020.

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv , 2021.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS , 2020.

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment. arxiv , 2024.

Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv , 2023.

Allan Jabri, David Fleet, and Ting Chen. Scalable adaptive computation for iterative generation. Proc. ICML , 2023.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. NeurIPS , 2022.

Junjie Ke, Keren Ye, Jiahui Yu, Yonghui Wu, Peyman Milanfar, and Feng Yang. Vila: Learning image aesthetics from user comments with vision-language pretraining. CVPR , 2023.

Diederik P Kingma. Auto-encoding variational bayes. ICLR , 2014.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In ICCV , 2023.

Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arxiv , 2023.

Andreas Koukounas, Georgios Mastrapas, Bo Wang, Mohammad Kalim Akram, Sedigheh Eslami, Michael Günther, Isabelle Mohr, Saba Sturua, Scott Martens, Nan Wang, et al. jinaclip-v2: Multilingual multimodal embeddings for text and images. arXiv , 2024.

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. NeurIPS , 2019.

Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, Meng Cao, and Yinfei Yang. Revisit large-scale image-caption data in pre-training multimodal foundation models. arxiv , 2024.

Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, and Cihang Xie. What if we recaption billions of web images with llama-3? arxiv , 2024.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Chase Lambert, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-image alignment with deep-fusion large language models. arxiv , 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS , 2024.

Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In ECCV , 2024.

Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. In Proc. ICML , 2020.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , 2023.

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer learning. arXiv , 2023.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe 358 Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image 359 synthesis. arXiv , 2023. 360

361

362

363

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proc. ICML , 2021.

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 2015.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. NeurIPS , 2022.

Sahand Sharifzadeh, Christos Kaplanis, Shreya Pathak, Dharshan Kumaran, Anastasija Ilic, Jovana Mitrovic, Charles Blundell, and Andrea Banino. Synth 2 : Boosting visual-language models with synthetic captions and image embeddings. arxiv , 2024.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. Proc. ICML , 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. ICLR , 2020.

Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. Communications of the ACM , 2016.

Yonglong Tian, Lijie Fan, Kaifeng Chen, Dina Katabi, Dilip Krishnan, and Phillip Isola. Learning vision from models rivals learning vision from data. CVPR , 2023.

Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV , 2021.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS , 2017.

Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In ICCV , 2023.

Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv , 2024.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. arXiv , 2022.

Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong, Jonathan Huang, Jinwoo Shin, 405 and Saining Xie. Representation alignment for generation: Training diffusion transformers is 406 easier than you think. arXiv , 2024. 407

Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon 408 Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In 409 ICCV , 2019. 410

A Additional Qualitative Results at 256 2 resolution 411

A.1 Based on ImageNet validation set captions 412

A.2 Based on DPG bench captions 413

DiT-I

B Additional quantitative results 414

C Implementation details 415

LayerNorm

Multi-head

Self-Attention

LayerNorm

Multi-head

Cross-Attention

+

LayerNorm

Pointwise

Feedforward

+

+

Time

Z-Norm

Text

Time

Text

Condition Block-1

Condition Block-2

LayerNorm

Multi-head Self-Attention

LayerNorm

Pointwise Feedforward

+

+

Condition Block

Cross Attention

From Condition-

ing to Latents

Cross Attention

From Pixels

to Latents

Self Attention

xN

Self Attention

x2

shared

across

RIN-I

blocks

Cross Atten-

tion From La-

tents to Pixels

Latents

Time

FLAN T5 XL

Embeddings

Latents

In this work, we use both DiT [35] and RIN [20] architectures. To adapt DiT for text-conditional 416

setting, we replace AdaLN-Zero conditioning with cross-attention to input the text condition into the 417

model, as in [3]. Before feeding the text condition to the model, we refine it using two self-attention 418

layers. Similar to [8], we add QK-Normalization [14] in each of the self-attention and cross-attention 419

blocks to mitigate sudden growths of attention entropy and reduce training loss instability. We also 420

add LayerScale [49] to each of the residual blocks of DiT for further stability. Figure 6 details our 421

DiT-I architecture. 422

To adapt the RIN [20] for the text-conditional setting, we used the off-the-shelf architecture from [6], 423

an adaptation of the RIN architecture detailed in the Appendix of [6]. Figure 6 details our CAD-I 424 architecture. 425

We use the framework of latent diffusion [40]. For encoding the images into the latent space, we use 426

the pre-trained variational autoencoder [23, 50] provided by the authors of Stable Diffusion [40]. 427

The checkpoint used is available on HuggingFace: https://huggingface.co/stabilityai/ 428

sd-vae-ft-ema . For text conditions, we encode the captions using the T5 text encoder. The 429

checkpoint is available on HuggingFace: https://huggingface.co/google/flan-t5-xl . 430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

D Captioning details

Captioning efficiently with LLaVA To caption images, we use the checkpoint llama3-llava-next-8b-hf (available on HuggingFace: https://huggingface.co/ llava-hf/llama3-llava-next-8b-hf ) with the prompt "Describe this image" . LLaVA encodes images using a dynamic resolution scheme. It processes both the entire image and four distinct patches as unique images and concatenates them. For 256x256 images, LLaVA uses around 2500 image tokens. To make the captioning process more efficient, we prune the image tokens, retaining only the tokens of the entire image and discarding patch-specific tokens. This optimization increased inference speed by a factor of 2.7, without compromising performances. Examples of long captions generated by LLaVA are given in Figure 7.

Captioning CutMix images We caption CutMix images from CM 1 / 2 with similar settings used for captioning the original ImageNet images. However, to ensure that LLaVA does not describe both the base and the CutMix images independently, we use a different prompt: 'Describe this image. Consider all the objects in the picture. Describe them, describe their position and their relation. Do not consider the image as a composite of images. The image is a single scene image' .

For settings CM 1 / 4 ,CM 1 / 9 and CM 1 / 16 , LLaVA tends to either ignore the smaller CutMix image or describe the image as a composite of two images. To avoid this behaviour, we encode the image by using the entire image patch and add tokens from the patch to which the CutMix image belongs. We use the following prompt: 'Describe this image. Consider all the objects in the picture. Describe them, describe their position and their relation. Do not consider the image as a composite of images.

The image is a single scene image'

images are given in Figure 7.

. Examples of long captions generated by LLaVA for CutMix

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

495

496

497

E CutMix details

The CutMix framework systematically combines concepts while preserving object centrality. Our framework defines four precise augmentation patterns, each designed to maintain visual coherence while introducing novel concept combinations. These are briefly described below:

1. CM 1 / 2 (Half-Mix):

Scale:

Both images maintain their original resolution. Deterministic split along height or width. Each concept occupies 50% of final image. Both concepts maintain full resolution.

Position:

Coverage:

Preservation:

2. CM 1 / 4 (Quarter-Mix):

Scale:

CutMix image resized to 50% side length. Fixed placement at one of four corners. 2nd concept occupies 25% of final image.

Position:

Coverage:

Preservation: Base image center region remains intact.

CM 1 / 9 (Ninth-Mix):

Scale: CutMix image resized to 33.3% side length. Position: Fixed placement along image borders.

Coverage: 2nd concept occupies 11.1% of final image.

Preservation:

Base image center, corners remain intact.

CM 1 / 16 (Sixteenth-Mix):

Scale: CutMix image resized to 25% side length.

Position: Random placement not central 10% region.

Coverage: 2nd concept occupies 6.25% of final image.

Preservation: Base image center region remains intact.

Each augmentation strategy generates 1,281,167 samples, matching ImageNet's training set size. Figure 7 shows examples of the different structured augmentations.

We also define CM all , which uniformly samples from all four patterns. The CM all variant combines equal proportions (25%) from each pattern to maintain the same total sample count. Postaugmentation, we apply LLaVA captioning to all generated images, ensuring semantic alignment between visual and textual representations. This produces detailed descriptions that accurately reflect the augmented content while maintaining natural language fluency.

E.1 Training with CutMix images

Because the CutMix image augmentations have strong artefacts corresponding to the boundaries of the mixing, we have to prevent the model from learning those salient features and reproducing them. To that end, we propose to train on image augmentation only at timesteps t where the noisy image x t is sufficiently noisy that the artifacts no longer matter. In practice, this corresponds to sampling either from the original image training set A or from the augmented image training set A IA conditionally to t , compared to an additional hyperparameter τ deciding whether t is sufficiently large for image augmentation. This extra condition leads to replacing the original diffusion loss in Equation 1 with

In this novel loss, the timestep t ∼ U (0 , T ) is still sampled uniformly. We introduce a new random 492 variable ρ that is sampled conditionally to t , where B τ,p ( t ) denotes a specific distribution that 493 corresponds to: 494

Here, B p a Bernoulli distribution of parameter p . The text-image pair ( x 0 , c ) is then sampled conditionally to ρ , where A ( ρ ) is a distribution that uniformly samples from the original or the augmented datasets depending on ρ :

Setting

LLaVA w/o CM

Long Caption

In the heart of the verdant background, a delicate white butterfly with a hint of yellow around its edges is captured in mid-flight. Its wings, a blend of two shades of cream and white, add a soft contrast to the vibrant colors of the flower it's hovering over. The flower, a stunning shade of purple, is adorned with a yellow center that speaks of the sun's rays it might have been basking in. Surrounding it is a lush green foliage, adding depth to this tranquil scene. The butterfly, positioned slightly to the left of the flower's center, seems to be in the process of alighting or perhaps taking off, adding a dynamic element to this otherwise serene tableau.

On the left side, there is a person playing the trumpet on a street. The individual is standing on the ground, wearing a long-sleeved shirt and jeans. On the ground next to the player, there is a trash can. The street is made of asphalt, with buildings in the background and a clear sky overhead. On the right side of the image, there are two penguins standing on a gravel surface, possibly a road or a rocky beach. The penguins are frozen in place, displaying a contrast to the dynamic scene on the left side. The penguins' yellow beaks, black bodies, and white feathers are clearly visible. The image captures an everyday moment of music played on the street, juxtaposed with the surreal and unexpected presence of the penguins. The image does not provide information about the time of day, specific location, or any actions being performed by the penguins or the person. The image is static, presenting two moments captured in time rather than a sequence of events.

The image shows a large, muscular dog in the foreground and a silver sports car in the background. The dog, appearing to be a Golden Retriever, is on the left side of the frame, occupying a significant portion of the space. It has a yellow hue with darker markings, and its ears are floppy. The dog is heavily coated with snowflakes, suggesting that the photo was taken in a snowy environment. The sports car, positioned on the right, is a two-door coupe with sleek curves and a notable design, featuring the Mercedes-Benz logo on its front grille. The car has a silver finish, and the photo captures it from a perspective that shows the front and side profile. The car is parked on an asphalt surface, possibly a parking lot or a driveway. The dog is facing the camera with direct gaze, while the car is positioned slightly towards the side, away from the viewer's perspective.

The image depicts a picturesque outdoor scene featuring an ornate building, which appears to be a palace or manor house, with classical architectural elements including symmetrical windows, a central cupola, and multiple chimneys. In front of the building is a well-maintained garden with pathways and neatly trimmed hedges or borders. Above the garden, there is a clear blue sky with a few scattered clouds. In the sky, there is a single hot air balloon with a bright orange and yellow pattern. The balloon is floating at a considerable height above the garden and the building, suggesting it might be part of a leisure activity or a special event. The image is a photograph with natural lighting, indicative of a sunny day.

The image is a photograph featuring a husky dog resting in the snow. The dog has a light coat with darker markings around its face and ears, and it is lying on its side with its head up, looking directly at the camera. Its eyes are open and its mouth is slightly open, showing teeth and a pink tongue, which suggests the dog might be panting or in a relaxed state. Next to the dog's side, there is a wine glass with red wine and a few purple flowers, which could be lilacs, positioned on the left side of the glass stem. The wine glass and flowers are set against a blurred background that gives the impression of greenery.

Figure 7: Long captions generated by our synthetic LLaVA captioner . The captions generated are highly diverse and add in much more intricate details of compositionality , colors as well as concepts which are not present in the original ImageNet dataset. The captions generated for our augmented images are also highly coherent and explain the scene in a much more realistic way.

504

Algorithm 1 Batch with CutMix image augmentation

CutMix

ImageNet

ImageNet CutMix

LLaVA

LLaVA

Long captions

Normal batch

Long captions

CutMix batch

t

p

t > τ

and

p > ρ

Long captions

Combined Batch

Denoising

Model

Data Curation

E.2 CutMix Augmentation Ablations

E.2.1 Ablation on CutMix settings 505

First, we analyse the performances of the pixel augmentations for { CM 1 / 2 , CM 1 / 4 , CM 1 / 9 , 506 CM 1 / 16 , CM all } settings. We fix the probability of using a pixel-augmented image in the batch 507 when t > τ to p = 0 . 5 and we measure both image quality and composition ability. Results are 508 509

reported in Table 9.

For image quality, all settings seem to perform similarly, with CM 1 / 2 being the best at 6.13 FID 510 and CM all being the worst at 6.81 FID. This indicates that all settings are able to avoid producing 511 uncanny images that would disturb the training too much. 512

For composition ability, CM 1 / 16 can improve over the baseline on extended prompts, whereas CM all 513

performances between the original prompts and the extended ones. Since CM all is a mixture of 515

can improve over the baseline on original prompts. Overall, only CM all manages to keep closer 514

all other settings, it also has the most diverse training set and is thus harder to overfit. As such, we 516 consider CM all for the best models. 517

E.2.2 Ablation on CutMix probability

Next, we analyse the influence of the probability p of using a pixel augmented image in the batch, when the condition on t is met. Results for p ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } are shown in Table 10, using CM 1 / 4 pixel augmentations.

518

519

520

521

522

523

524

525

As we can see in terms of image quality, the FID is slightly degraded by having too frequent pixel augmentation ( p > 0 . 5 ). This can be explained by the fact that pixel-augmented images are only seen when t > τ . As such, a high value for p creates a distribution gap between the images seen for t > τ and the images seen for t ≤ τ .

Composition ability shows a similar behaviour with the GenEval overall score decreasing when p 526 increases for both the original and the extended prompts. As such, we consider p ≤ 0 . 5 for the best 527 models. 528

E.2.3 Ablation on threshold τ 529

Finally, we analyse the influence of the threshold τ , which enables CutMix images to be sampled 530 in training batches. Table 11 shows the FID Inception on ImageNet Val and the GenEval scores of 531 models trained with different τ values. 532

We find that τ = 400 results in the highest GenEval score of 0 . 55 on original prompts and 0 . 57 on 533

extended prompts, while τ = 600 yields the lowest FID on ImageNet Val. As such, we use τ = 400 534

for the best models. 535

536

537

538

539

540

541

542

543

544

545

546

F Cropping details

Our cropping training methodology (see Figure 9) removes spurious concept correlations due to its masking scheme. We maintain the original captions and force the model to independently identify relevant textual elements. This creates a more challenging learning task for the model that enhances text-image alignment. During training, we only consider tokens corresponding to small portion of the image and mask out the rest from both the loss function and cross-attention layers. Given we do this online, this is highly efficient and also allows an infinite training set based on ImageNet to train on. For making the model understand the full dynamics of the training data, in our training scheme with crops, we only feed cropped images to the model with a probability, p = 0 . 5 . In rest of the cases, we use entire image. To keep the training scheme as simple as possible, for cropped versions, we only use crop resolution of >50% of the normal resolution.

ImageNet

LLaVA

LLaVA

Long captions

Normal batch

Long captions

Crop batch

p

p > ρ

Long captions

Combined Batch

Denoising

Model

Data Curation

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

G Qualitative results prompts

Here we show the prompts used to make the Figure 4. Note that for AIO we use the short version of the prompt as it is closer to its train distribution:

1. A pirate ship sailing on a streaming soup

The image showcases a colossal, exquisitely crafted pirate ship, its presence commanding and larger-than-life, as it sails triumphantly across a boundless sea of steaming soup. The ship's hull, made of dark, polished wood, is adorned with intricate carvings of dragons and waves, while its three towering masts support vast, billowing sails that glow faintly in the warm, golden light radiating from the broth. The soup is a vibrant, aromatic masterpiece, with swirls of rich broth, floating islands of noodles, and vibrant vegetables like carrots, bok choy, and mushrooms creating a textured, immersive landscape. The ship's deck is alive with detail-ropes coiled neatly, barrels stacked high, and a crow's nest peeking above the sails, all slightly damp from the soup's rising steam. The bowl, an enormous, ornate vessel, is crafted from gleaming porcelain, its surface painted with delicate, hand-drawn scenes of mountains and rivers, adding a layer of cultural richness to the surreal composition. The scene is both absurd and breathtaking, blending the grandeur of a seafaring adventure with the comforting, whimsical charm of a bowl of soup, creating an image that is unforgettable and endlessly imaginative.

2. A hedgehog and an hourglass

The image features a small, brown hedgehog with its characteristic spiky coat, standing near an hourglass in the middle of a dense forest. The hourglass is made of clear glass, and fine grains of sand are visible as they fall from the top chamber to the bottom. The forest surrounding the hedgehog and the hourglass is lush and green, with tall trees and thick undergrowth. Sunlight filters through the leaves, creating dappled patterns on the forest floor. The scene evokes a sense of tranquillity and the passage of time. The hedgehog appears to be observing the falling sand, perhaps contemplating the fleeting nature of time.

3. A teddy bear riding a motorbike

A plush teddy bear, adorned with a shiny black motorcycle helmet and a flowing red cape, is perched confidently on a miniature red motorcycle. The toy bike and its adventurous rider are positioned against the bustling backdrop of Rio de Janeiro, with the iconic Dois Irmãos mountain peaks rising majestically in the distance. The scene captures the playful contrast between the soft texture of the teddy bear and the sleek metal of the motorcycle, all under the bright Brazilian sun.

4. A teapot and some cookies

A detailed illustration of a teapot sitting on a decorative tablecloth, with delicate floral patterns and intricate stitching. The teapot itself has a sturdy handle and a gleaming silver spout, emitting a gentle steam as if freshly poured. The surrounding table features a few scattered tea leaves, and a plate with a few cookies, adding a touch of warmth and coziness to the scene. The illustration style is whimsical, with bold lines and vibrant colors, creating a sense of playfulness and inviting the viewer to take a sip from the teapot.

5. A goat on a mountain top

Adetailed photograph of a majestic goat standing atop a rocky outcropping, its white coat speckled with patches of brown and its curved horns reaching towards the sky. The goat's eyes are alert, and its ears are perked up, as if listening to some distant sound. In the background, a serene landscape unfolds, with rolling hills and a distant mountain range, all bathed in soft, warm sunlight that casts gentle shadows across the goat's fur.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly state the main claim of achieving competitive text-to-image generation results using only ImageNet with augmentations, reaching performances of larger models. This is supported by the experimental results presented in Sections 2 and 3

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses limitations (See Section 2 such as early overfitting due to the smaller scale of ImageNet and challenges in complex compositional generalisation with ImageNet's object-centric nature.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper's contributions are empirical, focusing on a novel training setup and experimental validation rather than new theoretical results.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper details the architectures used (DiT-I and CAD-I), the dataset (ImageNet), text and image augmentation strategies (TA, IA - CutMix, Crop), captioning details, and training procedures. Specifics about CutMix settings and the modified loss function are provided in the appendix.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer:[Yes]

Justification: The paper explicitly states that all the training data are hosted at https: //huggingface.co/datasets/anonymous_for_review and all the code and models are hosted at https://github.com/anonymous_for_review ." Appendices C, E, D, and E also provide implementation, captioning, and CutMix details.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper discusses the dataset (ImageNet), evaluation on ImageNet-50k validation and MSCOCO-30k. It also details model architectures, text and image augmentations, and scaling to higher resolution (Section 2. Appendix C provides further implementation details including the use of a pre-trained V AE and T5 text encoder with HuggingFace links. Appendix E provides details on CutMix settings, including the probability 'p' and threshold ' τ ' Appendix F provides details on Crop settings. While specific optimizer details (AdamW, learning rate schedule, etc.) are not explicitly in the main text, one could find them in the config files of the training code.

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper reports various metrics (FID, CLIPScore, GenEval, DPGBench, PickScore, Aesthetic Score, VILA Score) in tables. However, measures of statistical significance are not reported for these experimental results.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper states that the compute budget is about 500 H100 hours for one training. It also mentions that the high-resolution fine-tuning takes an "additional 100,000 steps". While specific memory or detailed per-experiment breakdowns are not extensively provided in the main text, the overall compute budget is mentioned.

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research focuses on text-to-image generation using a publicly available dataset (ImageNet) and aims to improve reproducibility and accessibility. The paper discusses potential societal impacts in the conclusion.

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The conclusion mentions positive impacts such as computational efficiency, opening possibilities for specialized domain adaptation. While direct negative impacts are not extensively detailed, the call for "more sustainable and responsible development" acknowledges the broader context.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper focuses on training models on ImageNet, a well-established and curated dataset, rather than scraped web data. While image generation models can have misuse potential, the paper's contribution is more on the methodology of training with open-source data and does not introduce new large models trained on high-risk scraped data. The provided models are relatively small (300-400M parameters).

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites ImageNet [41] as the primary dataset. It also mentions using pre-trained VAE from Stable Diffusion [40] and T5 text encoder, providing HuggingFace links containing license information. The LLaVA captioner [32] used for generating captions is also cited with a HuggingFace link. References are provided for architectures like DiT [35] and RIN [20].

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The models developed and the augmented training data (ImageNet with synthetic captions and CutMix augmentations) are hosted on HuggingFace and GitHub with complete documentation.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The research described in the paper does not involve crowdsourcing experiments or research with human subjects for data collection or evaluation.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve research with human subjects.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: The paper explicitly states the use of LLaV A [32], a Vision-Language Model (VLM) for generating synthetic captions for the ImageNet dataset. The details of the LLaV A model used and the prompting strategy are provided in Appendix D.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a training paradigm with data augmentation for T2I models using only the ImageNet dataset. The approach can be extended to generate high-resolution images, demonstrating that effective performance does not require massive datasets or models with billions of parameters. The proposed model achieves competitive results on benchmarks like GenEval and DPG-Bench, outperforming popular models such as SDXL and PixArt-$\\alpha$ despite having a smaller model size and training dataset.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n* The paper is well-organized and clearly written. The motivation behind the work is effectively communicated, with detailed explanations that guide the reader.\n* The topics addressed are timely and relevant, especially in the context of the growing scale of models and datasets.\n* The experimental setup is clearly presented, and the results are thoroughly explained, making the conclusions easy to follow.\n\n### **Weaknesses**\n\n* Although the authors aim to challenge the notion that \"bigger is better,\" the results still somewhat support this idea. For instance, in Table 6, models like SD3 and Janus outperform the proposed method in various aspects. While the proposed model achieves the second-highest overall score, Janus surpasses it in several specific categories. Additionally, techniques like data augmentation—used in this work—effectively expand the data space, indirectly reinforcing the idea that \"more is better.\" A tighter focus on data efficiency alone might strengthen the central message.\n* The proposed method is only evaluated on ImageNet, without testing on other widely used datasets such as LAION or DataComp. Applying the same training pipeline to a small, randomly sampled subset (e.g., 0.001B samples) from these datasets could help validate the generalizability and robustness of the approach.\n* The timing and conditions for applying CutMix are critical, yet these details are included only in Appendix E-1. Moving this information to the main paper would improve clarity and help alleviate concerns about potential artifacts introduced by CutMix.\n* Most of the provided images depict either single objects or multiple objects from the same class. It is recommended to include results showcasing more complex multi-object scenarios to better demonstrate the model’s text-to-image (T2I) generation capabilities.\n\n---\n\n### **Overall Evaluation**\n\nOverall, I lean toward accepting this paper. The contributions are timely and relevant, and the results are promising. However, the paper would benefit from a clearer articulation of its position with respect to the \"bigger is better\" narrative, as well as more comprehensive experimental details. Addressing these points would further strengthen the paper's impact and clarity.\n\nQUESTIONS:\n* Could the proposed approach be applied directly to other datasets to improve data efficiency more broadly? Given the known limitations of ImageNet, starting with a richer or more diverse dataset might lead to even better results.\n* Although CutMix is applied only during noisy timesteps, using images that are significantly different may still introduce artifacts. Are there any limitations in the current approach that might lead to such issues, and how are they addressed?\n* How does the model handle concepts that are rare or absent in ImageNet, such as abstract scenes, complex interactions, or human-centric prompts?\n* I’m also curious about how CutMix contributes to learning spatial relationships in complex scenes. Could the approach be extended to involve more than two objects during augmentation (three or four) to further enhance compositional capabilities?\n\n\n\n(Since the reviewer understands that training a T2I from scratch is time-consuming and might not be on time within the rebuttal timeline, it is acceptable for the authors to provide text-only explanations.)\n\nLIMITATIONS:\nWhile the authors acknowledge the limitations of using the ImageNet dataset, they do not discuss the limitations of their proposed approach. Here are some from the reviewer's perspective:\n1. The CropMix augmentation may disrupt visual aesthetics from a human perspective. A more refined augmentation method could be developed to enhance compositional ability while preserving visual appeal.\n2. Since the method relies on data augmentation from external models (e.g., text augmentation using LLaVa), establishing a validation protocol would help ensure the quality and accuracy of the augmented data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper challenges the \"bigger is better\" paradigm in text-to-image generation by demonstrating that models trained solely on ImageNet with well-designed text and image augmentations can match or outperform state-of-the-art models trained on billion-scale datasets. The findings are surprising and can benefit the GenAI research.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. By using ImageNet (1.2M images) with synthetic captions and augmentations (e.g., CutMix, Crop), the model achieves good performance. \n2. ImageNet’s standardization and open access address critical issues in the field (closed-source data, decay), enabling reproducible research. The training setup requires ~500 H100 hours, making it accessible to smaller teams.\n3. Metrics include FID, CLIPScore, GenEval, DPGBench, and aesthetic scores, covering image quality, text-image alignment, and human preference, have been reported.\n\nWeaknesses:\n1. The novelty is limited; I mean the idea is novel, while the paper does not propose any new approaches, the key approaches in the paper are re-caption and augmentation. The former one is proposed by DALL-E 3, and the last one has been widely deployed among computer vision tasks.\n2. ImageNet’s object-centric bias; The dataset lacks complex scenes (e.g., humans, environments), leading to limitations in generating diverse contexts. This is the limitation of ImageNet, because this is a 1000-class classification dataset. For better visualization, maybe you can have a minor part on discussing how to improve it, by adding more data from a large-scale dataset such as CC3M, JourneyDB, etc.\n3. The 512^2 resolution is too small for generation, although I understand generating 512^2 resolution images is very difficult if using Imagenet. Can you use a few number of 1024^2 samples to continue to scale up the resolution?\n\nQUESTIONS:\nHow to further improve multi-object composition without increasing data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to explore the potential of training text-to-image (T2I) models using only the ImageNet dataset, in an attempt to challenge the prevailing \"bigger is better\" paradigm that relies on massive, often private, web-crawled data. The authors propose a training pipeline combining synthetic captions (text augmentation) and specific image augmentations (CutMix, Crop). Through this pipeline, they claim that their lightweight model (~400M parameters) can match or even surpass SOTA models like SDXL, which are orders of magnitude larger in both parameter count and training data size, on specific compositionality benchmarks (e.g., GenEval). However, the paper's core premise, methodological purity, and the comprehensiveness of its evaluation are deeply flawed, leading to a misleading conclusion.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe motivation of the paper is clear, and the author provides two solutions, TA and IA, to close the performance inferior of training only on the ImageNet dataset for the T2V task. The methods are simple and achieve a reasonable gain.\n\n**Weaknessess**\n1. The writing needs to be improved. There are too many typos in this paper. The gray color highlight in the Tables is disordered. The nominations, such as \"DINOv2,\" have different font styles.\n\n2. The anonymous link provided by the author is empty, which cannot support reviewing the paper.\n\n3. The motivation of the proposed methods is clear and reasonable. However, the solutions are quite simple and lack novelty.\n\n4. The baselines SD v2.1 and SDXL are not the SOTA method in the current era. Modern baselines such as SD3 and FLUX are required.\n\n5. The effectiveness of the proposed methods is not robust across different models. For example, the CutMix is not effective on CAD-I as in Table 3. More fundamental methods need to be evaluated, such as SiT and traditional U-Net.\n\n6. Why are there no experiments for Crop on CAD-I? Does it achieve unsatisfactory performance or fail to converge?\n\n7. Training only on ImageNet may result in an overfitting problem during the longer training process, which could be found in Figure 3. Thus, in many T2V tasks, a larger training dataset is required to provide more diverse visual patterns and image-text alignment datapoints.\n\nQUESTIONS:\nMy questions are listed above in the weaknesses section. If the author could address my concern, I would prefer to raise the final score.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper challenges the prevailing belief that massive web-scale datasets (e.g., LAION-5B) are essential for high-quality text-to-image (T2I) generation. The authors propose a simple yet effective alternative: training T2I diffusion models solely on ImageNet, augmented with synthetic long captions and compositional image augmentations. They adapt the DiT and CAD architectures for this setup and demonstrate that models trained on only 1.2M images (i.e., ImageNet) can outperform or match large models like SDXL on GenEval and DPG-Bench using just 1/1000th of the data and 1/10th the parameters. Their approach emphasizes reproducibility, efficiency, and interpretability through open data and code.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengh\n\n### Reproducibility & Simplicity:  \nUses the widely available and standardized ImageNet dataset, making the approach reproducible and accessible without relying on proprietary or decaying datasets.\n\n### Strong Efficiency:   \nAchieves competitive or superior performance (e.g., +1% on GenEval, +0.5% on DPG-Bench) compared to state-of-the-art models with drastically less compute, data, and model size.\n\n### Well-Analyzed Augmentation Strategy:  \nCarefully investigates and validates the effectiveness of both textual (LLaVA-based long captions) and visual (e.g., CutMix, cropping) augmentations to overcome limitations of ImageNet.\n\n### Compositional Strength:  \nDespite the limited dataset, the proposed models exhibit impressive compositional generalization and prompt following, demonstrated both quantitatively and qualitatively.\n\n# Weakness\n\n### Lack of Comparison with High-quality (high aesthetic & high resolution & enriched caption) Dataset - LAION-POP [1]:\nThe paper does not compare its approach to models trained on high-quality datasets such as LAION-POP, which consists of high-aesthetic, high-resolution images with enriched captions generated by strong vision-language models like LLaVA. Unlike ImageNet—which primarily consists of low-resolution (≤512×512), object-centric, and realistic photos—LAION-POP supports both low- and high-resolution model training due to its consistently high-quality images. Recent work such as the KOALA paper [2] has demonstrated that, despite its relatively small size (~600K images), LAION-POP enables models to outperform those trained on larger-scale datasets (e.g., 2M images) in terms of visual quality and compositional alignment. Furthermore, in Line 254, the authors argue that their findings highlight the potential of smaller datasets to achieve state-of-the-art results, thereby enabling specialized domain adaptation where large-scale data collection is impractical. However, this claim is not entirely novel. The KOALA study has already shown that LAION-POP—a smaller but purpose-built dataset for text-to-image generation—can lead to competitive or even superior performance. This precedent challenges the necessity of using ImageNet for training text-to-image models and calls into question the novelty of using smaller datasets as a contribution. Without a direct comparison or justification for choosing ImageNet over more suitable datasets like LAION-POP, the generality and practical significance of the authors’ findings remain unclear.\n\n### Limited Evaluation of Aesthetic Quality: \nWhile the paper does include some quantitative aesthetic evaluation in Table 4 using metrics like PickScore, Aesthetic Score, and VILA, these automated measures alone are not sufficient to assess the nuanced and subjective nature of visual aesthetics in text-to-image generation. In particular, human perception of beauty, style, and realism often diverges from what these metrics capture. The absence of human evaluation therefore remains a critical limitation, especially given that aesthetic quality is a key criterion for practical T2I applications. Furthermore, the paper could be improved by incorporating more reliable automated metrics for aesthetic assessment, such as HPS (Human Preference Score) [1] and ImageReward [2], which have been shown to better align with human judgments. A human study—complemented by these stronger metrics—would provide a more comprehensive and interpretable evaluation of aesthetic quality.\n\n### Limited Practical Quality of Generated Images and qualitative comparison with SoTA  open-source models:\nWhile the authors claim that “smaller datasets can achieve state-of-the-art results,” the qualitative results presented in Figures 1, 4, and 6 suggest otherwise. Many of the generated images appear to lack fine-grained detail, visual coherence, or aesthetic appeal. This raises concerns about the practical usability of the proposed approach in real-world scenarios such as creative content generation, user-facing applications, or commercial design tools—domains where high visual quality is often a non-negotiable requirement. Moreover, the paper does not provide qualitative comparisons with recent state-of-the-art open-source text-to-image models such as KOALA, PixArt-series, or SANA-series. These models have demonstrated strong visual fidelity and compositional accuracy, even in challenging prompts. Without side-by-side visual comparisons, it is difficult to assess the relative strengths and limitations of the proposed method, especially given its reliance on ImageNet—a dataset not originally intended for generative tasks.\n\nIncluding such comparisons would help clarify the practical trade-offs between efficiency and quality and strengthen the authors’ claims regarding the viability of training high-performing generative models from smaller or more accessible datasets.\n\n---\n[1] https://laion.ai/blog/laion-pop/  \n[2] Lee et al. KOALA: Empirical Lessons Toward Memory-Efficient and Fast Diffusion Models for Text-to-Image Synthesis. NeurIPS 2024.  \n[3] Wu et al. Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis. Arxiv 2024.  \n[4] Xu et al. ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation. NeurIPS 2023.\n\nQUESTIONS:\n1. What resolution was used when generating images with SDXL for comparison? The SDXL samples in the paper (e.g., Figure 5) appear to be of relatively low quality, which raises concerns that the model may have been run at 512×512 resolution. Since SDXL is trained and typically evaluated at 1024×1024, using a lower resolution could significantly degrade output quality and lead to an unfair comparison.\n\n2. Comparison with Recent State-of-the-Art T2I Models:\nThe paper compares its models against SDXL and PixArt-α, which are relatively strong baselines. However, both of these models do not utilize enriched captions during training, which may explain their weaker performance on compositionality-focused benchmarks like DPGBench. Could the authors provide comparisons against more recent models that do use enriched or multi-level captions, such as PixArt-Σ (Sigma), SANA, or FLUX? Including these would provide a more thorough and up-to-date evaluation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}