EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models
Authors: Rui Zhao
Hangjie Yuan
Yujie Wei
Shiwei Zhang
Yuchao Gu
Lingmin Ran
Xiang Wang
Jay Zhangjie Wu
David Junhao Zhang
Yingya Zhang
Mike Zheng Shou
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation
Large Vision-Language Models
Diffusion Models
Primary area: diffusion_based_models

Abstract: Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available at https://github.com/showlab/EvolveDirector.

EvolveDirector: Approaching Advanced Text-to-lmage Generation with Large Vision-Language Models
Rui Zhao', Hangjie Yuan', Yujie Wei2.3, Shiwei Zhang", Yuchao Gu', Lingmin Ran Xiang Wanga4, Zhangjic , Junhao Zhang', Yingya Zhang", Maanng,,Men Mike Zheng Shou
Show Lab, National University of Singapore
2 Alibaba Group
3 Fudan University
Huazhong University of Science and Technology
Abstract
Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The code and model weights are available wtwpg/hhww...ooooooo//////eeeeee
Introduction
In the field of Al-generated content, an increasing number of advanced models have showcased their ability to generate realistic and imaginative images, such as Imagen [1], DALL-E [2], Stable
Diffussin 3 [3]. Midjourney [4]. While these models benefit from publicly available datasets such as ImageNet [5], LAION [6], and SAM [7], they rely more heavily on proprictary, high-quality data collections that surpass the quality of publicly accessible datasets. Models such as Midjourney [4] are particularly noted for deriving substantial benefits from their internal datasets. However, given the significant commercial ad advantages brought by their impressive capabilities, most advanced models keep their parameters private, hindering reproducibility and democratization. In this paper, we aim to explore training open-source text-to- text-to-image model with public resources to achieve comparable capabilities with the existing advanced models.
"Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

1
2
3
front pymmmidss, Spider-Man.
right, Batman,
Three urispprrett glass bottles on weoden cable.
The on the left has red liquid and the number number The one on the right has green liqud and the number 3.
Pallet knife technique, large intentional brosiistr mastespiaee painting leaaad silently prowling through underbrush, fantaay.
In gomlen, there sits white cat on the left and black dog on the right, with green water placod between the two
NEURIPS 2024
capricating book cover festures a young woman with enchunting pare, dressed in vibeant attire, suroundedd swarling colors,
a world of chacs and beauty.
A slock and moden logo design for NaurIPS 7024 The text "NeurIPS 2024* writen in bold, black capital letters In the background, there's minimalist, abstract parts highligheed coles, The design clem, visually
representation
appeoling, and instantly reoognzable.
volemic cription dense jungle creates striking acare as lave flows down the mountain, contrasting with the lish folings. Dark sky, casting landscape.
move-capped mountain, a silver riboon that connects the forest and canyons.
way, munsuring flowing geatly, resembling
Images generated by our model Edgen (ExolveDirrecor-Gee) Edgen can generate high- quality images with multiple ratios and resolutions. Notably, it excels in generating text and avoiding attribute confusion when generating multiple objects, which significant characteristics of the most advanced text-to-image models available today. The input text prompts are annotated under the corresponding images.
Figure 1:

---

Despite the fact that internal data and model parameters of many advanced models remain inaccessible, they provide publicly accessible application programming interfaces (APIs) that enable users to access their generated distribution. This leads to the construction of synthetic benchmarks, e.g., JourneyDB [8] collects 4.7 million images generated by Midjourney [4]. This benchmark is further utilized for enhancing the training of new generative models [9]. However, this paradigm is not data efficient, posing challenges in terms of substantial computation and expenses. Instead of statically constructing multiple expensive large-scale datasets for each advanced model, we take a step forward in this paper by delving into recovering their generative capabilities a unified framework with limited samples. We propose EvolveDirector to address this challenging task by shedding light on two research questions: (1) How many synthetic text- eext-iaae pairs are sufficient to approximate the generative capability of an advanced model? (2) Taking it step further. is it possible for the base model to obtain generative capabilities beyond the advanced models?
To explore the first question, we start with demonstration experiment by training a relatively poor model, a DiT model [10] pre-trained on public dataset ImageNet [5] and SAM [7], to approach the advanced model PixArt-a [9] using increasing data scales. The training data is curated by collecting diverse text prompts and utilizing them to generate images from PixArt-a. The experiments indicate that when we scale the training data (i.e., generated image-text pairs) to 11 million, we can obtain a base model achieving similar capabilities to the target model without access to its internal data. However, the magnitude of 11 million generated data comparable to the 14 million data used for pre-training the target model. Training a base model in this way incurs significant expenses, not only in terms of time and computational resources but also the costs associated with using fee-basee APIs of some advanced models.
For more efficient training, it is crucial to minimize data redundancy and maximize data quality, as the marginal benefit of training on inferior data is limited. The corpus of the 11 million text prompts, generated from the SAM dataset [9], and the images generated by the advanced model exhibit redundancy in several aspects: (1) lacking imaginative text prompts due to the photographic nature of SAM images; (2) high similarities among text prompts; and (3) imbalanced data quality. The generated images using the target advanced model may exhibit low quality due to inferior alignment some text prompts. To address these problems, we introduce large vision- vision-language models (VLMs) to improve the diversity and quality of training data for efficient training. Our approach involves continuous evaluation with VLMS to dynamically refine the training dataset by the discrimination, expansion, deletion, and mutation operations. This dynamic curation strategy ensures that only valuable data is retained, significantly redueing the volume of data for training. Experimental results demonstrate that a mere 100k training samples are sufficient for the base model to gain similar performance to that of the target model, which substantially fewer than the 11 million samples required by the baseline method.
To explore the second question, we applied EvolveDirector to train the base model to approach multi- ple recent most advanced models in a unified framework, including DeepFloyd IF [11], Playground 2.5 [12]. Stable Diffusion 3 [3], and Ideogram [13]. For each text prompt, we invoke advanced models to generate their images, and the VI.M selects the best match to train our base model. The final trained model is named Edgen (EvolveDireetorr-Ce Gen). The experimental results demonstrate that Edgen outperforms the advanced models mentioned above. Although our initial goal was to approximate these advanced models, we ultimately benefited from the VLM in choosing better training samples from their generated data, thereby achieving capabilities superior to any individual model.
The code of ExolveDirector and the model weights of Edgen are released to benefit the downstream tasks. Our contributions are summarized as: (1) Through experiments on massive data, we conclude that the generation abilities of text-to-image model can be approximated through training on its generated data. (2) We propose the EvolveDirector, a framework that harnesses VLM to direct the training of the base model to learn generation ability from advanced models efficiently. (3) The trained text-to-image model Edgen outperforms the most advanced models.
2 Related Works
Text-to-Imaae Generation. To advance the overall quality of text-to-image generation, research efforts have been invested in exploring architectural improvement [10, 14, 15, 16) and generation paradigm advancement [17, 18, 3], etc. Diffusion models stand out as the de facto text-to-image
3

---

generation paradigm [19, 20, 1, 21, 22, 23, 24, 25], noted for its scalability and stability [26]. They benefit numerous downstream tasks, spanning image editing [27, 28]. video generation [29. 30, 31, 32], 3D content generation [33, 34], etc. Through the use of highly descriptive and aligned image-text pairs at a substantial scale, text-to-image models that excel resolutions, safery control, and the capability to render accurate scenes are obtained, e.g., Imagen [1]. Midjourney [4], DALL-E 3 [2]. Stable Diffusion [3], and Ideogram [13]. However, the exceptional capabilities of most advanced models have led providers to restrict access, typically offering only APIs, which limits their widespread and equitable use. In this paper, we aim to fill this gap by leveraging advanced VLMs to direct base model to replicate the functionality of advanced models. In contrast, some works propose to motivate models to learn from their self- geferaeeed images [35,66].
Evaluating T21 Generation with VLMs. Some automatic evaluation methods [37, 38, 39] are propoesd to combine the L.LMs and VQA models to evaluate the generated contents. Thanks to the substantial advancement of large language models [40, 41, 42, 43, 44, 45), the capabilities of VLMs are largely boosted [46, 47, 48, 49, 50, 51]. Utilizing these enhanced capabilities, methods building on VLMs are designed to facilitate the evaluation of text-to-image generation [52, 53, 54, 37, 55]. Notably, the recent research effort, Gecko [54], demonstrates the practicality of leveraging pre- trained LLMs [44] and VLMs [45] for systematic evaluation of text-to-image generative performance, spanning aspects such as text rendering, relational generation, attribute generation, etc. However, previous research requires fine-graine teeegraaddd evaluation across various aspects, which remains challenging. In EvolveDirectorr we simplify this process by requiring only pairwise comparisons, which enables a stable and reliable performance, facilitating the approaching of advanced text-to-image models.
Knowledge Distillation. KD [56] aims to transfer knowledge from well-trained teacher models to a simpler student model. The primary focus of most research in KD lies in investigating distillation losses with the output predictions of teacher model [57, 58J, intermediate feature maps [59, 60], or feature correlations [61, 62] to distill knowledge. Recently, distillation methods based on diffusion models have garnered attention, primarily by distilling outputs from intermediate steps of the dif- fusion process to expedite the sampling process [63, 64, 65, 66]. Despite sharing the same goal of approaching the performance of advanced models, we would like to highlight our training paradigm is an orthogonal procedure to KD. To approach the advanced models with only APIs available, avoid the need for distillation losses or acquiring intermediate results and instead choose a data-driven paradigm. Furthermore, our designed paradigm is also orthogonal to dataset distillation [67] evaluate and refine data during training rather than relying on a pre-existing large dataset.
Online Learning. Online learning has garnered significant interest for its capacity to enable models to adapt to real-time and dynamic data scenarios [68, 69J. This attention extends to a variety of real-world tasks, including semi-supervised learning [70, 71J, unsupervised learning [72, 73], and continual learning [74, 75, 76), etc. In EvolveDirector, we harness the potential of online learning and powerful VLMs to evolve models towards advanced generation models, offering efficient and scalable framework capable of dynamically adapting to evolving data.
3 Method
In this section, we outline the EvolveDirector framework See. 3.1, describe the detailed operations of the VLM within this framework in Sec. 3.2, and discuss the training strategies in Sec. 3.3.
3.1 Overview of EvolveDirector
The proposed framework EvolveDirector, as shown in Fig. 2, comprises three parallel processes: (I) interacting with advanced T2I models to get training images, (2) maintaining the dynamic training set empowered by the Language Model (VLM), (3) training the base model on the dynamic training set. The dynamic training set is updated by the VLM and advanced T2I models, ensure the data are high value for training, thereby achieving efficient training and reducing the overall required data volume.
Interaction with Advanced Models. While the model configurations and weights of many advanced T21 models are not publicly accessible, they often provide APIs for interactions. In EvolveDirector, we interact with these APIs by submitting text prompts and receiving the corresponding generated images. The one that aligns with the given text prompt better will be selected by the VLM and

---

Users
Imput
(a) Interactions between community users and advanced T2I model..
Advanced T21 Models
Ouipul
Generated Images
Internt with available
Inpul
Output
(b)
Training
Inference
A
EvolveDirector
Base T21 Model
Text Prompls (dymamic sel)
Training Images (dynamic set)
Generated Images
Expend, Delete,
Vision- Language Model x
Discriminate
Mutate
is better
Updkited Text Promplsi
The overview of the proposed framework EvolveDirector. (a) Advanced T2I models provide accessible APIs, allowing users imput text prompts and get the generated images. (b) The base model is trained on the dynamic dataset, consisting of text prompts and corresponding images generated by advanced models via API calls. The VLM continuously evaluates the base model and, according to its performance, dynamically updates and refines the dataset through discrimination, expansion, deletion, and mutation operations based its evaluations.
Figure 2:
included in the training set. The selection criteria and details of these advanced models are elaborated in Sec. 4.4.
Dynamic Training Set. The training set is dynamically updated during the training of the base model. continuously incorporates high-value samples, i.e. text prompts on which the base model underperforms compared advanced models. Simultaneously, it dynamically excludes low-value samples, i.e. those on which the base model performs comparably to the advanced models. The VLM evaluates the value of samples, with a detailed procedure outlined in Sec 3.2. The advanced T21 models are continuously called to generate new images for the new text prompts and update them into the dynamic training set.
Training of the Base Model. Based on the dynamic training set, we train a Diffusion Transformer (DiT) [10] architecture proposed by PixArt-a [9]. Besides, we incorporate layer normalization after the Q (query) and K (key) projections in the multi-head cross attention blocks [77] to further stabilize the training, Attention(4) Q, K, V) = softmax (1a(9)) (f((Q)-xx())" normalizations after the projections.
base text-to-image model. Specifically, the model is built upon the improved
V. where fo(+) and (-) are the layer Vil
3.2
Vision-Language Model as Director
The VLM acts as a director to guide the construction more valuable dynamic datasets. To simplify the task difficulty and better leverage the capabilities of VLM, present it with choice of two images as a multiple-choice question, as shown the top left corner of Fig. 3. One image Ladvanced is generated by the advanced model, while another one /aase is generated by the base model, respectively, In practice, the order of the two images is randomized. VLM called to compare them and decide which one aligns better with the given text prompt T. Regarding the choice of VLM, there two potential scenarios as follows.
(1) Tastanned outperforms Trase- The inferior performance on this text prompt indicates that the base model is still under-trained. Therefore, ExolveDirector utilizes the VLM to generate more Nşg variations of this text prompt T. as shown in the lower-left corner of Fig. 3. Then the advanced T2I
5

---

model generates corresponding images to expand the dynamic training set, as shown in the right side of Fig. 3. The original samples will continuously be involved in the training.
(2) Iadvanced does not outperform Thase- If the base model is comparable with the advanced model, indicating sufficient learning, the VLM will remove that prompt I' low-value sample from the set to economize on training resources.
Besides the expansion and deletion operations, EvolveDirector also performs mutation operations with certain probability. This operation permits the VL. to generate more diverse text prompts in- dependent of any existing ones, thereby encourag- ing the model to explore and learn from a broader domain of text prompts.
triloo Ohrend (80, wand onchoee Sutur and desctie Eoastee seitan
acoanaapptt fook atull?
You lave tou sptan -(AI Ioiiat,
compasio aulcnilyss
(Beer des Iauts langgs depii fagn wootherk
3.3 Training Strategies
(A) weutie beitei and asae that nouprlie mare lae descrppa "ander.
Online Training. To boost training efficiency, we develop EvolveDirector as an online training framework. Specifically, the base model under- goes uninterrupted training, without pausing for the advanced model or the VLM to execute. In- stead, it sends command to start VLM evaluation every 100 epochs, termed a checking epoch. At the checking epoch, a subset of text prompts is sampled from the dataset with a specific ratio Rg. They are fed into a replica of the base model generate images. Then the VLM evaluation and the subsequent execution of EvolveDirector be- gin, as illustrated in Fig. 3. Finally, a command sent to the trainer of the base model to update the dynamic dataset. A detailed analysis of hyper- parameters, including select ratio Rs and exten- sion number Ns, is provided in the supplementary.
Reylien the neme the wer disccptionn with ather loack objics, chunccses backgooak acd tioweptions
"A redivvll liight
gelleping Jiwn wullleacen stree is cads
deatists bovatalk
veserall suales"
avrroust invanur
regget
donilas
An example of the interaction between the EvolveDirector, VLM, and advanced T21 model. For brevity, auxiliary instructions to the VLM are omitted in this figure.
Figure 3:
Stable Training. In task setup, the scale of training data is significantly less than the million scale. Under this circumstance, the original architecture [9] demonstrates considerable instability during training, and the generation collapse is observed. Thus we follow the work [77] and apply the layer normalization after the query and key projections improve the training stability. The experimental results detailed in the supplementary demonstrate the effectiveness of this adaptation.
Multi-Scale Training. The ability to generate images across various scales and aspect ratios is significant capability of advanced T2I models. To facilitate more efficient training, we initially train the base model on images with a fixed resolution of 512px. Subsequently, we extend the training to images of higher resolution with multiple aspect ratios, thereby enabling the model to generate multi-scale and mulli-ratio images. Following [9], we construct buckets with different aspect ratios and image sizes. EvolveDirector, for each text prompt, a size bucket randomly sampled from the buckets and advanced T2I models are called to generate images with this size. To avoid generation collapse, if the selected bucket falls outside the optimal size range of the advanced model, it will be resized the closest appropriate size.
4 Experiments
4.1 Training Details
We train the base model on 16 A10 GPUs for 240 GPU days, with batch size of 128 and 32 for images at 512px and 1024px resolution, respectively. The VLM evaluation process distributed across 8 A100 GPUs to facilitate its speed. The open-source advanced models are deployed on our

---

devices with simulated APIs for interaction. For closed-source models, EvolveDirector interacts with them through their public APIs.
4.2 Selection of Vision-Language Models
Table 1: Alignment of VLMs with Human Prefer-
ences. The best value is highlighted in blue and the second-best value is highlighted in green
The ability of the VLM to analyze images is crucial to the EvolveDirector framework. There are multiple powerful VLMs available, includ- ing CogVLM [82], CogAgent [78]. Qwen-Vi VL (Qwen-VL-Chat, VL-Chat, Qwen- VL-Plus, and Qwen- VL-Max) [42], In [79], LLaVA and LLaVA-Next [51, 80], and GPT-4V [81]. We evaluate their newest version on 600 pairs of questions to caleulate the alignments with hu- man raters. Each question consisted of text prompt and two images generated based on that prompt, presented to 5 different human raters. The VLMs are tested in two aspects: (1) Discrimination, and (2) Expansion. To be more specific, (I) initially, the VLMs were required to select which image aligned more closely with the given text prompt. The output is scored by O for wrong or for correct by human raters. (2) Subsequently, they are instructed to generate more variations of the given text prompt. The score of "Accuracy" evaluates whether the generated text prompts contain any linguistic emros and whether they in the same syntactic structure as the given text prompt. The score of "Diversity" evaluates the diversity among the generated text prompts. These two scores range from (worst) to 5 (best). The results are shown in Tab. 1. The LLaVA-Next and GPT-4V achieve the top performance. Considering that VA- is totally free to use, we select it as the VLM for EvolveDirector.
Expansion Accuracy Diversity
Discrimination
Cog-Agent [78]
0.675
3.81
3.76
Qwen- VL--Mxx [42]
0.825
4.72
4,56
InternVL. [79]
0.793
4.70
3.67
LLaVA-Next [80]
0.840
4.85
4.69
GPT-4V [81]
0.847
4.82
4.72
4.3 Ablation Studies
Table Ablation Studies. The best value is high- lighted in blue and the second-best value is high- lighted in green
Models. For ablation studies, we select the Pixart-a [9] as the unified advanced model to approach. We utilize a DiT model pre-trained on publicly accessible data, i.e. the ImageNet and SAM dataset, the base model.
Discrimination Expansion Mutation
Data Scale
L Evaluation
x
11M
x
X
IM
11.49
39.44
Metrics. We sample 10, 000 text prompts to feed into models trained under different ablation settings to generate images and calculate their FIDs with the images generated by the advanced model. These text prompts are not seen by these models in the training stages. To conduct human evaluation, randomly selected 300 sets of text prompts and their corresponding generated images, which are 2, 400 in total. These images are paired in twos, each consisting of one image from the advanced model and one from ablation model corresponding to the same text prompt, arranged in random order. Then they are presented to 5 different human raters choose which image matches the text prompt better. In this manner, each ablation model is paired with the advanced model for comparison, and their selected ratios are recorded to reflect their relative performance compared to the advanced model.
x
X 100K
15.19
32.22
100K
x
X
100K
13.05
36.44
X 100K
7.61
47,00
100K
7.45
48.53
Results. The results of FIDs and human evaluation are shown in Tab. 2. (1) Directly Training on Generated Data. The first three models were directly trained on images generated by the advanced model, using image quantities of 10 million, 1 million, and 100 thousand respectively. The experimental results show that the model trained on 10 million data reaches comparable level to the advanced model in terms of human preference (48.89% V.S. 51.11%), and achieves a low FID score (7.36). This indicates that the generative capabilities of the advanced model can be learned through training on its large-scale generated data. However, if the training data is reduced to 10% or even 1% of the original amount, the performance of the trained model will significantly decrease.
(2) Training with EvolveDirectorr In the last four rows of Tab. 2, models trained with EvolveDirector under different ablation settings are evaluated. The upper bound of data volume is set to 100k for all models. The first model trained on an initial number of 100K data and dynamically deletes initial number of 2K data and dynamically
data. The last three data models start training
7

---

add new data to learn. The first model applies the "Discrinination " function of the VLM model to discriminate the generated samples of the base model. Samples comparable to the output of the advanced model are removed from the training dataset. The results show that dynamically deleting samples does not cause much performance degradation. The second model does not use the VI.M to evaluate the base model but instead randomly selected training samples for "Expansion", i.e. generating more variants of given prompts and training samples. The results show that this operation brought a slight performance improvement due to the expansion of text prompt diversity. The third model utilizes VLM to evaluate the base model and performs reasoned expansion and deletion based on the evaluation results. As the results indicate, there is a significant performance increase, which highlights the importance of the evaluation of VLM. Lastly, the complete version of EvolveDirector is tested, which further applies the Mutation, i.c. randomly generating entirely new text prompts with a 10% probability. This operation further improves the performance of the model, because it encourages the model to explore more diverse images. With the full version of EvolveDirectoor, the model trained on a dynamic dataset with a data cap of 100K, achieved performance comparable to the model trained on 10M generated data, indicating that the proposed framework could significantly reduce the amount of training data required to approach the performance of the advanced model.
It is worth noting that there is still a slight gap between the final model and the advanced model, which can be attributed to the law of diminishing returns. This occurs because it becomes increasingly diffi- cult to identify truly high-value samples as the performance approaches that of the advanced model. However, this phenomenon vanishes when EvolveDirector learns from multiple advanced models simultaneously, as experiments in Sec. 4.4 demonstrated. This is because the larger performance gaps between multiple models facilitate the easier identification of high-value samples.
4.4 Approaching Advanced Models
Table 3: Select Ratios of Advanced Models. The highest value is highlighted in blue and the low- est value is highlighted in gray
We select several latest advanced models ap- proach their powerful generation ability, includ- ing Playground 2.5 [12], Stable Diffusion [3], and Ideogram [13]. Playground 2.5 is famous for its aesthetic generative effects, while Stable Diffusion 3 and Idcogram are known for their strong performance in various aspects including text generation and multi-object generation. Be- sides, we select relatively old model, DeepFloyd IF (but just released in April 2023) [11], for its amazing text generation ability. The base model is the same as the one in Sec. 4.3, and we continue to train the base model that has already approached the Pixart-a to approach the selected multiple advanced models. During training, EvloveDiretcor will feed each text prompt to all of them to generate corresponding images, and then use VIM to select the best one of them as the image from the advanced model. The selected image then undergoes evaluation detailed in Sec.3.2.
Specific Domain
Overall
Human
Text Multi-Object
DeepFloyd IF [11]
18.57%
9.12%
21.21%
12.12%
Playground 2,5 [12]
25.71%
31.33%
9.09%
25.24%
Ideogram [13]
29.52%
29.18%
34.24%
33.36%
Stable Diffusion 3 [3]
26.19%
30.36%
35.45%
29.27%
Select Ratios of Advanced Models. We have calculated the proportions of images generated by these models that were selected by the VLM in the training stage, as shown in Table 3. Among the generated images, those generated by Ideogram are selected with the highest ratio. Besides, we evaluate the select ratios of images in specific domains, such as human generation, text generation, and multiple object generation. Examples of these three types are shown in Fig. 5. The results in Table 3 show that different advanced models achieve the highest select ratios in different specifics. For example, for generating text in images, the Stable Diffusion outperforms others, while the Playground 2.5 is much worse than others. This may be caused by that the internal training data of Playground 2.5 does not include sufficient high-quality images with text in them. This also demonstrates the significance of diverse high-qualit data.
Quantitative Comparison. To evaluate the performance of the trained Edgen, the base model, and the advanced models, we sample text prompts and feed them into each model to generate images. One image generated by Edgen and one image generated by the comparison model are combined together. Finally, same with the scale of comparison in previous works [9], 300 text prompts and 1800 image combinations are shown to human raters to select which one aligns with the given text prompt better. Each combination is evaluated by 5 different human raters. The results are shown in Fig. 4. We first analyze the performance of each model across all tested text prompts, as shown on the left side of Fig. 4. It is noteworthy that during the training, the VLM selects the best ones among
8

---

Oruel
Houes Grensie
Mell- Julk-OhjetGnervtte
84pm
Made
Mdpx
54p
Pland
Deafloc 3f
14pm
34pm
1Apu
Plyyoodd 15
5Age
Subla Ditecse)
TAgm
46
89
100
100
20
68
90
100
20
90
Figure Human evaluation of the images generated by the base model, Edgen trained by the proposed EvolveDirector, and multiple advanced models.
"A scasoned journalist, his charismatic face framed by shelves of archived stories and reports. His eyes, sharp and scrutinous, glance over the rim of his glasses at the historic headlines that surround him."
MAKE
6
Mal TOURRN
TODAY MORZIIG
MAZININ
cheerful mug on cluttered office desk, side emblazoned with Make Today Amazing! vibrant, uplifting colors."
"A small wooden boat drifling calm lake, containing young Ghild wearing life jacket, playful puppy with its tongue and calm old cal dozing off beside them."
Playground
Stable Diffusion
Base Model
Edgen (ours)
Pixart-a
DeepF/loyd IF
Ideogram
2.5
EvoleDDrrcctor
Advanced 721 Models
Figure 5: Images generated by the base model, Edgen trained by our EvolveDirector, and multiple advanced models. The results in three rows showcase the generation of human, text, and multi-object.
the images generated by multiple advanced models be used training data. Therefore, although EvolveDirector initially aims to train the base model to approach them, the final trained model Edgen outperforms all of the advanced models. Besides, evaluate the performance of these models on specific types of text prompts, with the results displayed on the right side Fig.
Qualitative Comparison. In Fig. 5, we showcased three groups of generated images. The three rows of results respectively demonstrate the generation capabilities for humans, text, and multiple objects. For the first group, only the images generated by Edgen, DeepFloyd IF, and Ideogram successfully reflect the "face framed by shelves Stable Diffusion 3 have the ability generate correct text in images. For the results shown in the third row, three objects need to generated, i.c. the child, puppy, and cat. Only Edgen and Idcogram success and other models lost some objects. These results show that Edgen has already learned powerful generation abilities and outperforms some advanced models.
As shown in the second row, only Edgen, Ideogram, and
5 Conclusion
In this paper, we propose EvolveDirector, framework that targets approaching the generation capabilities of advanced text-to-image models by only utilizing their publicly accessible APIs. By harnessing the capabilities of large vision-language models for evaluating image-text alignment,

---

EvolveDirector significantly reduces the volume of training data required, thus saving considerable training costs, especially those associated with API usage. Experimental results demonstrate that the resultant model Edgen, inheriting the generation capabilities from multiple advanced models, achieves superior performance in various aspects. The limitations and future work are discussed in the supplementary.
Acknowledgements
This project is supported by the Mike Zheng Shou's Start-Up Grant from NUS. It was partially done by Zhao Rui during his internship at Alibaba Group, and Zhao Rui is partially supported by the Alibaba Research Intern Program.
References
[1] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36499 2022.
[2] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dallle- pdf. 2(3):8, 2023.
[3] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transform- for high-resolution image synthesis. arXiv preprint (tXXXx::40330200066 2024.
[4] Midjourney. Midjoumny. https: / /twp://www.//.oo///o mirneurreyyy.oooom
ourney com, 2023.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Ieee, 2009.
[6] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 252944, 2022.
[7] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[8] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, et al. Journeydb: A benchmark for generative image understanding. Advances in Neural Information Processing Systems, 36, 2024.
[9] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426, 2023.
[10] William Peebles and Saining Xie. Scalable diffusion models with transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[11] Stability AI. Deepfloyd if. https://githu: wtt.://ww...cceeeeeee//////////e 2023.
[12] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation, arXiv preprint arXiv:2402. 17245, 2024.
[13] Ideogram. Idcogram. https: //te://g/dd/ommm//// wwt.gaamm..a.aaaaa..ma.aa. 2024.
10

---

[14] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCA. 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234-241. Springer, 2015.
[15] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[16] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[17] Tero Karras, Samuli Laine, and Timo Aila. style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4401 2019.
[18] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[19] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. Raphael: Text-to-image generation via large mixture of diffusion paths. Advances in Neural Information P Processing Systems, 36, 2024.
[20] Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, al. Emu: Enhancing image generation models using photogenic needles in a haystack. arXiv preprint arXiv:2309. 15807, 2023.
[21] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.013 324, 2022.
[22] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao. Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint wrXiv:2403.046992 2024.
[23] Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden and Saining Xie. Sit: Exploring flow and diffusion-basee generative models with scalable interpolant transformers. arXiv preprint 2rXiv:2401.087440 2024.
[24] Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint 240. 2024.
[25] Xin Ma. Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03045 2024.
[26] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794 8794, 2021.
[27] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426-2435, 2022.
[28] Chenlin Meng, Yutong He, Yang Song, Jiaming Song. Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108. 01073, 2021.
[29] David Junhao Zhang. Jay Zhangjic WU, Jia-Wei Liu, Rui Zhao, Lingmnn Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023.
[30] Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of rext-to-video diffusion models. arXiv preprint ar arXiv:2310.0846 2023.

---

[31] Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia- Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Videoswap: Customized video subject swapping with interactive semantic point correspondence. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7621-7630, 2024.
[32] Yujie Wei, Shiwei Zhang. Zhiwu Qing. Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan. Dreamvideo: Composing your dream videos with cus- tomized subject and motion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6537-6549, 2024.
[33] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall, Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 248,::2200.14988,, 2022.
[34] Rui Zhao, Wei Li, Zhipeng Hu, Lincheng Li, Zhengxia Zou, Zhenwei Shi, and Changjie Fan. Zero-shot coxt-to-paramett translation for game character auto-creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21013-21023, 2023.
[35] Jiao Sun, Deqing Fu, Yushi Hu, Su Wang. Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Hermann, Sjoerd van Steenkiste. Ranjay Krishna, al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. In Synthetic Data for Computer Vision Work- shop@ CVPR 2024, 2023.
[36] Jialu Li, Jaemin Cho. Yi-Lin Sung, Jachong Yoon, and Mohit Bansal. Selma: Learning and merging skill-specific text-to-image experts with auto-generated data. arXiv preprint 24Xiv:2403.06955, 2024.
[37] Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[38] Jaemin Cho. Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint 1000::2310.18255, 2023.
[39] Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Jun- hao Zhang, Wei Liu, Yuchao Gu, Rui Zhao, et al. Towards better metric for text-to-video generation. arXiv preprint arXiv:2401.0778/, 2024.
[40] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[41] OpenAL. GPT-4 technical report, 2023.
[42] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16669, 2023.
[43] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
[44] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403 2023.
[45] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint
2022.
12

---

[46] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[47] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[48] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint ar0iv:2304. /0592, 2023.
[49] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
[50] Gemini Team, Rohan Anil, Sehastian Borgeaud, Yonghui Wu, Jean- Bean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
[51] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances neural information processing systems, 36, 2024.
[52] Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, and Idan Szpektor. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36, 2024.
[53] Yujie Lu, Xianjun Yang, Xiujun Li, Xin Eric Wang, and William Yang Wang. Limscore: Unveiling the power of large language models in text-to-image synthesis evaluation. Advances in Neural Information Processing Systems, 36, 2024.
[54] Olivia Wiles, Chuban Zhang, Isabela Albuquerque, Ivana Kajić, Su Wang, Emanuele Bugliarello, Yasumasa Onoe, Chris Knutsen, Cyrus Rashtchian, Jordi Pont-Tuset, et al. Revisiting text- to-image evaluation with gecko: On metrics, prompts, and human ratings. arXiv preprint arXiv:2404. 16820, 2024.
[55] Jaemin Cho. Yushi Hu, Roopal Garg, Peter Anderson, Ranjay Krishna, Jason Baldridge, Mohit Bansal, Jordi Pont-Tuset, and Su Wang. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation. arXiv preprint arXiv:2310.182.. 2023.
[56] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprini 1503:::503302531
[57] Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 201 8.
[58] Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191-5198, 2020.
[59] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1163-9171,2019,
[60] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3779-3787, 2019.
[61] Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2339-23483 2020.
13

---

[62] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 3967-3976, 2019.
[63] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01 2023.
[64] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint urXiv:2202.00552, 2022.
[65] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
[66] Xiang Wang, Shiwei Zhang, Han Zhang, Yu Liu, Yingya Zhang, Changxin Gao, and Nong Sang. Videolom: Video latent consistency model. arXiv preprint arXiv:23/2.09109, 2023.
[67] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv: 18/1. 10959, 2018.
[68] Steven CH Hoi, Doyen Sahoo, Jing Lu, and Peilin Zhao. Online learning: comprehensive survey, Neurocomputiig, 459:249-289, 2021.
[69] Alexander Rakhlin, Karthik Sridharan, and Ambuj Tewari. Online learning: Random averages, combinatorial parameters, and lcarnability. Advances in Neural Information Processing Systems, 23, 2010.
[70] Islam Nassar, Munawar Hayat, Ehsan Abbasnejad, Hamid Rezatofighi, and Gholamreza Haf- fari. Protocon: Pseudo-label refinement via online clustering and prototypical consistency for efficient semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11641-11655 2023.
[71] Salah Ud Din, Junming Shao, Jay Kumar, Waqar Ali, Jiaming Liu, and Yu Ye. Online reliable semi-supervised learning on evolving data streams. Information Sciences, 125:153-171, 2020.
[72] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Yew-Soon Ong, and Chen Change Loy. Online deep clustering for unsupervised representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6688-6697, 2020.
[73] Qi Qian, Yuanhong Xu, Juhua Hu, Hao Li, and Rong Jin. Unsupervised visual representation learning by online constrained k-means. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16640-16649, 2022.
[74] Yujie Wei, Jiaxin Ye, Zhizhong Huang. Junping Zhang, and Hongming Shan. Online prototype learning for online continual learning. In Proceedings the on Computer Vision, pages 18764-18774, 2023.
the IEFF/CVF International Conference
[75] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputing, 469:28-51, 2022.
[76] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non-stationary data streams. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8250-8259, 2021.
[77] Mostafa Dehghani. Josip Djolonga. Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[78] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.0891 2023.
14

---

[79] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Internv:: Scaling up vision foundation models and aligning for generic visual-linguisti tasks. arXiv preprini arXiv: 2023.
[80] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023.
[81] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[82] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0307, 2023.
15

---

A Limitation and Future Work
While EvolveDirecttor achieves significant strides in approximating the generation capabilities of advanced text-to-image models, the resultant model can face challenges related to bias. Our model can inadvertently inherit biases present in the images generated by the advanced models. Additionally, our reliance on VLMs to evaluate generated images could introduce their own biases into the selection process. To mitigate these issues, is better integrate debiasing methods and incorporate human feedback in future developments, aiming to enhance the robustness and faimess of EvolveDirector.
B Broader Impacts
Our research leverages the foundational capabilities of large lision-language models to reproduce advanced text-to-ine ext-to-image models. This initiative represents a pioneering approach to make state-of- the-art generative technologies more accessible and cost-effective. Our methodology substantially reduces the volume of data required for model training, thereby reducing computational demands and minimizing environmental impacts associated with image generation. The resultant model Edgen has the potential to revolutionize the creation of digital media owing to its inheritance of capabilities from multiple advanced models. For instance, its capabilities enable the creation of diverse media content and marketing materials, such as logos, by effectively rendering text into realistic and imaginative visual representations.
However, we also acknowledge potential negative societal impacts. As with many generative technologies, there is inherent risk of bias in terms of gender and race in the generated content. Furthermore, the improper use of the resultant model, such as inputting harmful or obscene text prompts creating unauthorized or deceptive images of public figures, could facilitate misinformation or impersonation. To mitigate these risks, it is imperative to implement thorough oversight to secure its ethical use.
C Layer Normalization
As discussed in the main paper, due to the scale of training data being significantly less than the previous million scale, the original architec- ture [9] is not stable during training. As shown in Fig. A (a) and (b), when the training steps increase from 0.5k 2k, the main object in the generated image begins to be destroyed. To ad- dress this problem, we incorporate layer normal- ization in the multi-head cross-altention. To be more specific, the layer normalization added after the Q (query) and K (key) projections. This can significantly stabilize the training of the base model, Training model with the layer normaliza-
(a) Trrining 0.5k; win layer normolization after QK projections
Training staps:
Training stepe: 28; wi layer necmalization after QK projections
win layer nomtllzationn after QK penjections
Figure A: Illisturation of the impact of incorporat- ing layer normalization after QK projections on generation.
tion can avoid generating destroyed images, as shown in Fig. A (c).
D Detailed Instructions to VLM
Instruction for Discrimination. We input the combination of two images and instruct the VLM to choose which one aligns with the given text prompt better. The instruction as follows, "In these two images (A) and (B), which one aligns better with the text description "text prompt"? You have two options: <(A) better>, <(B) better>. Simply state your choice, no need for detailed explanation."
Instruction for Expansion. The instructions for expansion follows, "Replace the nouns in the text description: "text prompt" with other kinds of objects, characters, backgrounds, names, colors, or styles to generate extend num more diverse text descriptions. Arrange them in the format of list T"Text description 1", "Text description 2", 2"...."."
Instruction for Mutation. The instructions for expansion are follows.
16

---

"Now, exercise your imagination to generate one new text description for visual content that is completely unrelated to the previous images. It should have completely different structure from the previous descriptions. enhanced prompt. Arrange it in the format of list just like ""xxxx"]]"
The enhanced prompt is randomly sampled from the following ones:
"It should be rough and short."
"It should contain less than 30 words and be highly detailed."
"It should contain over 30 words with different granular levels of detail."
"It should contain over 50 words with a lot of details."
Structure the Outputs of VLM. The diversity of output formats from VLM can pose challenges for automated parsing. We found that by providing specific instructions to the VLM, its output format can be standardized. Specifically, when prompting the VLM to generate more text prompts, offer instructions such as "Arrange them in the format of a list I "TTxx description 1", "Text description 2", ...J." This approach directs the VLM to generate outputs in a consistent format.
E Hyper-parameters Setting
The hyper-paramedee parameters of EvolveDirector include the select ratio of images selected from the dynamic training set for evaluation (select ratio Rs) and the number of training samples expanded based on single sample (number of extensions N).. These two parameters mainly affect the growth rate of training samples in the dynamic training set. Too fast growth rate can lead to the need to generate a large number of images in the early stages of training, thereby quickly increasing training costs, while too small an increase can cause the model explore the diversity of different samples too slowly. To balance the training costs and the diversity of samples explored by the model, it is necessary to set reasonable values for Rs and NE. Due to the high cost of searching for the most reasonable parameter combination (fully training the model under each combi- nation requires 240 A100 GPU days), we only explored limited number of parameter combinations and stopped the training as long as we observed the trend in data growth.
Training Stops vs Namber of T'aining Samples
16000
Rs - 10%6 Ne -3
20%, Nr-
14000
Rş = 50%, Ne 3
Rs 20%, Ne 11
Rs 20%, Ne -5
Number of ofTiinngg Samples
12000
10000
6000
4000
2000
2000
4000
5000
8000
10000
Training
Figure B: The effect of values of hyper- parameters on training data growth rate.
As shown in Fig. B, five kinds of combination of select ratio Rs and number of extensions NF are evaluated, we can observe that if the Rs is too large, the volume of training data increased rapidly with increasing cost too fast. Despite larger number extensions NE encouraging the model explore new training samples greedily, it also increases the training cost rapidly. In practice, we found that setting it to 3 enough to explore new samples. Finally, we select the combination of Rs = 20% and Ng = to achieve good balance of training cost and exploration of diverse samples. Besides, since the mutation rate R& does not affect the growth rate of the training set significantly, we simply set it to 10% an augmentation of the exploration of samples.
F
Implementation Details
The model trained by the Adam optimizer with a learning rate of 2e - and with a gradient clip of 1.0. A constant learning rate schedule with 1000 warm-up steps is used. The gradient checkpointing is used to save the VRAM. For generating 51 512px images, we train the base model with 100K training steps, and for 1024px image generation, train the model with 20K training steps.
The default number of training samples at the beginning of training is 20K, which will gradually grow to the upper bound. For the ablation studies in Sec. 4.3, we set the upper bound to 100K to explore the extreme performance of EvolveDirector: The initial text prompts are sampled from those captioned for the SAM dataset [9]. For approaching multiple advanced models in the Sec.4.4, we set
17

---

Space
"pear" nukdas the theae ooler ordzz wome a diewiig and iesiatt 10 realer of sevenct attng potiomencd ranuaar play. steez lecsepe ofeigg al plane, oet with caapee conaari jarnie garden
chanccer She artliig bla yeugg
hare koy hie has floee diice in heidded pigrass teikgrooal
ted weme reteiy wiir opicnes The dhenoree wocutig der chade parple polk painting
some The cremail and conepptal or unto eusatcon caprus
count whomnnn tharntie Sciendar
phangratyy
chieaas, vinses
seecol paecid dlieriiily sroms tuagel baly was the caack scune rathrs oerne beastull wis wam pertet
Acersen paok ridigg perplin
Kamning exocap flrncr stricdd denakd bateetly epes pasks. The Conar, wat in yirid caless and
shipe.
ofthe. gold
benatyy, with urding's buettfll
concig unniay rumer brage of the wos and degenls
qeaat wids coarars andin the thoners ritt hase The bockgoond sevee
greens moninng
saase sbll teak of delc hears in sees leowelll
castiiing rmeg
crenats willit phangraty. dak famayy
rcbeaa opperuiiia Blustratie teatehapol doc, banding coker caass: by
bantsaaed
oderld tadgmani purrat vitch bocking ouckad famand laraza shat 1y50g unticcal instigane: web.
inspamndllllkk Indgrmannt The wort cliad is playfa pefal sit's whlaee deiige. hegonner
iluttake, 24 ranies.
Figure C: Additional results 1/3, Images generated by the Edgen.
the upper bound to 300K to ensure more sufficient learning of much powerful generation abilities of the most advanced models. The initial text prompts are randomly selected from both the SAM captioning dataset and the community-souree text prompts.
G Additional Results
18

---

A stange high-defation plato ose wearng rakermmed glancs and engressd in his smartshens, atting za the todee The beltroon was decoraed fony postes, inclating one men laeghing
ookaffl coni-styll lt linseut ilas zise of captinding par ars rupired compaiioo, fimall model walting on pesk camatee nanayy she wenttu suicas, off- sheelder with dryyed docgg thv expour pama setian model abettact and mundia
The heed of lia with mejesty and elegaree. Clii Ligh and pride. itr her long velersincca, and ands you can see unth light and firz camin out of her aas. In tockgrant you can Senorrch with
goldes illmiazinn invting Bhrogi open stmoenkere Ponte plants floor light steaning bring touch Dooknes cheenti Sk image.
hody
boode
hemmoious and visually stiking soone.
Thatane deleat scene whnce wnall, aniiendy deniled hummingtte witt a spaikedd thac dees: hovers sex clater of lange, sull pirk flowen The eentng dhamlica,
duarning and optivating Illotaaioo of Malese pappy wannng a colorte, enduuring wints tat, atland platteous gudea party under the monnigs The sappy uniling enjoying woen atrrggurss armrrdd Iadgrarrd
eraniig ond whirsical image of bur aubicdoed ganny hoin zarhore under smail uniraias. A hege speen boves above them, pourise shower suger over the gamry bars, enarigg case and playti dvairwape,
muy. oongeroedd
budigrand saggreing
bookgoord heght deiglaad
paazl
detala oveill aismophere with nuch fanyy tile enduniment.
and
cotfidaat allutig genturran, undiing dewn stres Unoed degart NIX
shooing playfil and frienly
eveding dramatic clfft elhoartre
with wazping sul that cudes loy, The badkgrood Ecaturis only hone wath wame ficyloo,, santy strd and coterial panden, ilhatratioz.
thick for misdreevmn
accentamul by roft glow
fill mone. The
beos resng ses with lowering waxes croking
sease of one and danger The dk a rish Mexd doe base and pusto, with the moss cataur a dismmniing rellcction water. The evenal atmoghere case yet mostaiius, plvtn,
grous witsosin
Figure D: Additional results 2/3. Images generated by the Edgen.
19

---

Majolk slate-rtaned stilien, powcifd stasee, atricate pokken pated amme, mothical sum, timingg ruse and tai. repil posture, deniled amms musiius cloper and fenee dyremit poss, dranare lighting ceeting suddwa, fienry fantay illxaration, lighrosolation digtal
Ashiiing and dysanic 3D render Moust Plessuz High Scnsol's tye mases, wakeng dateminadly towaeds the viever the nitt sice of the sccen The tizzr, tua nawee, ferce exprosiie undes newnnt giving aff stregh
Cloc-up petroi, the vaterolat loss drarry yourg ley winh pairmn blunds her udtrndd with woath of
dinkliiea. Hiss arrons oxmarptaire expciian catreyys some of irmopection milenchll The boys cimplo
tiger bockgoond showess vikean sky cload
haiegrond.
creating wiking
cature
cloass, atmoephers. surral and upilting plon, 36 rasler.
light cnions desiba Queniin stake,
Abero Virga, Ziealow Beliiedi painting. dark Sreacy, Elntattion anire concpuua on, inchieeuee
Astarning beads on piz capturiig contal sonse in the lonopozas, meticdisaly painted sagull is pasded os reck, in winzs oussestcted and zyos fooe on
serenc and ronentic image of woraa with long becen bair and gasss, samee at arped beach chan. Siz weas yelone offohoalder dress that corplementa ode demestor aamy right da wirkiing
unticshoslly carved weeden odief Lepieting litll ons gecydoodic pupple and cun beagle.
Sotpiats golden anathos daneel
diapper the shoce,
citecape
contenplaion.
will
baok Enutatiin of
flosing Boooc
apploneee and a mortie idsed,
Delizaz flster softly vitrnt ganken blooiing flowes. Be warn snile idiatt sie stulls dnoegh nogcaal leasen, stagin of's Giittale soath 33953166, iltataation cosopuaa art
pless edeette amoorio, fiwe-to-zninn
windows ovcloding the city, and sokal starcnsc leading is ttelodisoo.
Figure E: Additional results 3/3, Images generated by the Edgen.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Claims in the abstract and introduction accurately reflect contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in Sec. A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the clear framework structure Sec. and training and implemen- tation details in Sec. 4.1 and Sec. F.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[Yes]
Justification: The code and model weights are open-sourced, and the link is attached at the end of the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CoodeSubmiis
httppscccppublic//eec S ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the detailed experimental setting in the front of each subsection of the experiment section, i.e. Sec. 4.2, Sec. 4.3 and Sec. 4.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiments we do do need error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //tuuriurip.../
Answer: [Yes]
Justification: We have carefully reviewed the Code of Ethics and strictly adhered to it in our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We claim social impact the appendix Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The safeguards are checked by the advanced models internally.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
The code, data, and models are all properly credited and their license and terms of use properly respected in work.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "waUR3P8zUk", "reviewer": "Reviewer_gzrb", "summary": "This paper explores the effectiveness of training a text-to-image (T2I) model using synthetic examples generated by existing T2I models. The authors find that on the order of 10M image-text pairs are necessary to approach the quality of a good model like PixArt-Alpha, while using only 1M or 100k examples results in serious deterioration.\n\nTo improve sample efficiency, the authors introduce their online learning framework, EvolveDirector, which leverages a vision-language model (VLM) to curate a better online training set. The VLM is used to compare the current trained checkpoint's generated images against the reference T2I model's images, in order to determine which training examples to drop from the training set, and which training example prompts to generate more variants of for further training. Occasionally, entirely new training prompts are introduced to increase diversity. Overall, this results in a training process which matches the quality of PixArt-Alpha according to human comparisons using only 100k examples.\n\nThe authors perform further ablations to show the necessity of each part of the framework, and train a model using four different advanced T2I models as targets, resulting in a final T2I model, named Edgen, which surpasses each of the target models in overall human preferences.", "questions": "1. What is the source of your training data? The training images are generated by the expert/advanced T2I models, but where do the input prompts come from?\n2. Following Q1, how do you know the seed prompts for training (i.e. the initial training set) cover a diversity of prompts? While using LLMs to create variations of prompts can result in new combinations of objects or concepts, the resulting new prompts are unlikely to capture the same diversity of text present in natural datasets.\n3. Where does the *mutation* operation source the new prompts from? Are they also generated by the same VLM, or taken from the same source as your initial training data? If they come from an existing dataset, what is the benefit of adding them midway through the training process instead of just starting with a initial training set?\n4. To my understanding, LLaVA-NeXT (and many other current VLMs) are not trained on examples involving multiple images. Are the images for comparison concatenated or passed separately to LLaVA? Do you notice any statistical confounders, such as choosing the first image more often than the second?\n5. The instructions for \"expansion\" to the VLM, to generate new variations on the prompt, specify only to replace nouns in the prompt. Why is this the strategy used to generate similar prompts? What happens if the T2I model is struggling to generate a specific noun, and the syntactic structure of the prompt is not the weak point? In that case, it feels like the new prompts would not be relevant for further training.\n6. When selecting which VLM to use, you say \"The output [of the VLM] is scored by 0 for wrong or 1 for correct by human raters\". Why not ask the human raters to rank the generations themselves, and then compute how many examples the VLM and the humans agreed on? Having the human raters rate the VLM output seems like it could induce bias in the raters.", "limitations": "The authors note that their method could result in a T2I model which inherits biases from both the teacher T2I model and the VLM used for evaluation. They also describe the potential positive and negative broader societal impacts of the research, including improving model training costs and potential for misuse.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality.** The paper demonstrates the possibility of a training framework using synthetic data which leverages expert T2I models and VLMs to maximize data efficiency. This opens up an avenue of research into better use of expert models for informing the training procedure in T2I.\n\n**Quality.** In certain areas, the authors perform thorough ablations to determine the best procedures, and to provide concrete numbers to compare against. For example, they evaluate the candidates for an expert VLM along several axes to justify their choice, and train baseline T2I models at several data scales to demonstrate the performance drop with decreasing scale.\n\n**Clarity.** The high-level approach and framework are explained clearly.\n\n**Significance.** The paper shows that existing T2I model performance can be replicated using only about 1% of the data needed with naive knowledge distillation.", "weaknesses": "**Quality.** While most of the experimental details are explained thoroughly, the paper omits any description of the text data used. The text data is a crucial part of the proposed framework, since it is used to seed the training data, as well as for the mutation operation, presumably. There is also no mention of the source of the data used for evaluation. Along the lines of evaluation, the dataset used to compute FID score should also be mentioned.\n\n**Clarity.** The paper fails to clarify many useful details earlier in the paper (abstract + introduction), leaving the overall understanding of the method vague until reaching Section 3. For example, the key components of expansion, deletion, and mutation are mentioned but not explained at all in the introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition and for providing detailed comments. We will address your concerns as follows.\n\nNotations: W: Weakness, Q: Question\n\n-----\n\n**W.1 & Q.1**: \nWe will incorporate more details about the source text prompts in the revision. Initially, the text prompts are randomly selected from both the SAM captioning dataset, aiming to capture the diversity of natural data, and the community-sourced text prompts, to ensure alignment with user preferences and interests. The ratio of samples from these two resources is equal to each other. These text prompts will be made available for public access. For evaluation and FID calculations, the text prompts are also sampled from these two sources and exclude the ones sampled for training. Below are some randomly selected examples of the text prompts.\n\nSAM Captions: \n\n- \"A woman and a young child, likely a toddler, standing together in a dimly lit area. The woman is wearing a headscarf, and the child is wearing a striped shirt. The woman is holding the child close to her, possibly providing comfort or protection. The overall atmosphere of the image is warm and intimate, showcasing the bond between the woman and the child.\"\n\nCommunity-sourced text prompts:\n\n- \"Text, \"Deisy Yaneth\" in one line, one word, no spaces, creative colorful sign made of watter and smoke, black background, artistic,ultra realistic, splash effect, incredible details, rich colors, high quality, in the style of Dan Mumford and Daniel Merriam, 3D rendering, poster, photo, fashion, cinematic, typography, 3d render\"\n\n- \"A realistic cat, act as a nurse, give a injection to me with syringe\"\n\n**W.2**: \nThanks for this comment. We will add more details and explanations in these sections.\n\n-----\n\n**Q.2**: \nAs mentioned in the response to W1&Q1, at the beginning of training, we sample the text prompts both from the SAM captions and community-sourced text prompts, to ensure a broad coverage of prompt diversity. Based on this, the VLMs are encouraged to generate more diverse text prompts, such as the example shown in Fig. 3. Besides, as shown in Table 1, the human raters agree with the high diversity of the generated text prompts. It is worth highlighting that there is no strict need to capture the distribution of text prompts in existing datasets, such as the SAM and LAION, where the long tail distribution, etc., may not be the most beneficial for training. Our proposed method aims to adaptively generate the samples that are high-value for training.\n\n**Q.3**:\nWe agree with you that if new prompts come from an existing dataset, there is no extra benefit. When developing our method, we took this into consideration and designed the mutation operation to create completely new text prompts using the VLM, without relying on existing ones. The mutation is operated in the training process to encourage the model to explore and learn from a broader domain of text prompts.\n\n**Q.4**: \nWe concatenate the two images to form a single contrast question input for the VLM. In our concatenating and comparing manner, as shown in Table 1,  LLaVA-NeXT achieves a similar performance as the Qwen-VL-Max, which is able to interact with multiple images. We arrange the two images in a random order to avoid any potential bias in selection. In our statistical validation, no location bias was found in the selections of LLaVA-NeXT. The results in the \"Discrimination\" column in Table 1 also show that the selections of LLaVA-NeXT match human raters closely.\n\n**Q.5**: \nPlease note that the expansion is to generate more text prompts and will not remove the original ones. If the T2I model is struggling to generate images aligned with the text prompts, the original text prompts will continuously be involved in the training. Besides, whether the new prompts are relevant for further training is determined by the VLM based on their value for learning. \n\n**Q.6**: \nWe agree with you that ranking the generations by human raters is an alternative way for evaluations. But in practice, human raters not only evaluate the discrimination, but also evaluate expansion accuracy and diversity, and the latter two are evaluated by scoring outputs of VLM. To streamline the evaluation process and ensure a consistent execution way for human raters, we choose the evaluation approach outlined in the paper. To eliminate potential bias, each output of VLMs is independently evaluated by five human raters, and the types of VLMs are anonymous to them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A3B7BudD0l", "reviewer": "Reviewer_i5a7", "summary": "This paper investigates how to train the text-to-image model comparable to advanced models using publicly available resources. Specifically, EvolveDirector collects training data with the APIs of advanced models, and further uses a VLM to continuously refine the training dataset. The proposed VLM refinement significantly reduces the data volume needed to teach a base T2I model and improves the training efficiency.", "questions": "Typos: in line 336, “outperform” -> “outperforms”.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed framework is novel, and can significantly reduce the volume of data required to approach the SOTA text-to-image generation performance.\n\n2. This paper investigates one interesting direction: training the base T2I model to approach the advanced models using their generated data, which can be crucial for bridging the gap between public T2I models and close-sourced T2I models. \n\n3. The way of utilizing VLMs is interesting and effective. The VLM dynamically maintains the training dataset to achieve efficient training.\n\n4. Edgen shows good human evaluation performance on generation faithfulness to text prompts, especially in multi-object generation and text generation.\n\n5. The paper is well-organized and technically clear.", "weaknesses": "1. The explanation of how to instruct VLM is not detailed. For example, how to ensure the generated text prompts can be parsed correctly and free of errors, which could introduce noise into the training data.\n\n2. It would be better to also evaluate the final model on other benchmarks (e.g., DSG, TIFA).\n\n3. Missing references: \n(1) DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback\n(2) SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your acknowledgment and comments. We will address your concerns as follows.\n\nNotations: W: Weakness, Q: Question\n\n-----\n\n**W.1**: \nThe detailed instructions for VLM are provided in the supplementary. We structure the outputs of VLM to ensure the generated text prompts can be parsed correctly and free of errors. The diversity of output formats from VLM can pose challenges for automated parsing. We found that by providing specific instructions to the VLM, its output format can be standardized. Specifically, when prompting the VLM to generate more text prompts, we offer instructions such as “Arrange them in the format of the list [\"Text description 1\", \"Text description 2\", ...].” This approach directs the VLM to generate outputs in a consistent format. \n\n**W.2**: \nThanks for the suggestion. TIFA [1] and DSG [2] are automatic evaluation methods designed to call LLMs to generate several questions and utilize the VQA model to answer them, so as to evaluate the alignment of the generated samples with the text prompts. Since DSG is the newest benchmark and more comprehensive. We evaluate the models on DSG and the results are reported as follows. The open-sourced mPLUG-large [3] is selected as the VQA model.\n\n|                             |          |                        |        |                        |       |                        |       | \n|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|                             | Base Model | Pixart-$\\alpha$ | DeepFloyd IF | Playground 2.5 | Ideogram | Stable Diffusion 3 |  Edgen (ours) |\n| DSG (↑)       | 68.61       | 72.60                | 74.49              | 74.98                   |79.97                | 80.05             |              80.61              |        \n\n**W.3**:\nThank you for providing these references. We will discuss these works in the related works section to strengthen it. These two works both aim to motivate T2I models to learn from their self-generated images. DreamSync [4] is proposed to improve the T2I models by selecting their own generations and fine-tuning them on the selections. SELMA [5] utilizes the LoRAs to fine-tune the T2I model on different skill-specific image-text pairs, and then merges these LoRAs to build a unified model. \n\n**Q.1**: \nThanks for pointing this out. We will do our best to check and correct the typos in the paper.\n\n----\n[1] Hu, Yushi, et al. \"Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Cho, Jaemin, et al. \"Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation.\" arXiv preprint arXiv:2310.18235 (2023).\n\n[3] Ye, Qinghao, et al. \"mplug-owl: Modularization empowers large language models with multimodality.\" arXiv preprint arXiv:2304.14178 (2023).\n\n[4] Sun, Jiao, et al. \"Dreamsync: Aligning text-to-image generation with image understanding feedback.\" Synthetic Data for Computer Vision Workshop@ CVPR 2024. \n\n[5] Li, Jialu, et al. \"SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data.\" arXiv preprint arXiv:2403.06952 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pl4wLTTB5w", "reviewer": "Reviewer_9FQn", "summary": "The paper introduces \"EvolveDirector,\" a novel framework designed to train a text-to-image generation model that can compete with advanced models using only publicly available resources. The authors aim to address the limitations posed by proprietary data and the inaccessibility of parameters in state-of-the-art models by leveraging public APIs to obtain text-image pairs for training.\n\nThe core of EvolveDirector relies on the interaction with advanced models through their APIs to generate data for training a base model. A significant challenge is the need for a large volume of data (10 million samples or more), which is both time-consuming and computationally expensive. To mitigate this, the authors employ pre-trained large vision-language models (VLMs) to dynamically update and refine the training dataset through various operations like discrimination, expansion, deletion, and mutation.\n\nThe framework's efficiency is demonstrated through experiments that show a significant reduction in the required data volume, with only 100k samples needed for comparable performance to models trained on 11 million samples. Furthermore, the paper claims that their final trained model, \"Edgen,\" outperforms several advanced models and will be open-sourced for downstream tasks.\n\nThe paper is well-written and presents a strong case for the capabilities of EvolveDirector. It would be beneficial to see additional experiments or comparisons with other state-of-the-art models to further establish the framework's superiority. Moreover, the paper could delve deeper into the specific mechanisms of knowledge transfer from the VLMs to the base model.", "questions": "See weakness.", "limitations": "Yes, this paper has limitation section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's core strength lies in its innovative approach to overcome the limitations posed by proprietary data and models. By leveraging publicly available APIs and pre-trained vision-language models (VLMs), EvolveDirector offers a feasible solution to access advanced text-to-image generation capabilities in an open-source environment.\nThe trained model, Edgen, shows impressive performance, outperforming several advanced models in various aspects. This speaks to The paper includes extensive experiments and ablation studies that validate the framework's effectiveness. The use of human evaluation for model comparison adds a layer of qualitative analysis to the quantitative results.\nThe authors acknowledge potential biases in the generated content and the importance of safety. They propose integrating debiasing methods and human feedback, showing a responsible approach to AI development.\n The methodology is technically sound, with a clear explanation of the framework's components and the role of VLMs in guiding the evolution of the base model. The paper also details the training strategies and hyperparameter settings, contributing to its reproducibility.\nThe paper discusses the broader impacts of the work, including potential positive societal impacts such as revolutionizing digital media creation and negative impacts like the risk of bias and misinformation. This demonstrates a well-rounded consideration of the technology's societal effects.", "weaknesses": "1. The layout needs to be adjusted. For example, Figure 1 takes up an entire page.\n2. The research question 1 & 2 is insightful, which is an important research topic in the perspective of distillation and data annotation.. However I see limited analysis or exploration for this question. Only one result from the trained DiT model to PixART may not be enough as the answer to question 1. As the authors motivate this work with question 1 & 2, introducing more related analysis would be helpful.\n3. It seems like the data construction pipeline could be improved by introducing images in reality as [1]. Given that you use VLMs for data refinement, it may be beneficial to use images in reality as ground truth and use VLMs to generation prompts.\n4. In Section 4.3, models trained without Discrimination & Expansion & Mutation (first line) can gain comparable results as models trained with Discrimination & Expansion & Mutation (last line). This may cause confusion, as it may lead to a questions: would it be necessary to perform Discrimination & Expansion & Mutation? Would it be all we need to just query advanced generative models to get data?\n\n[1] Zhao, Haozhe, et al. \"UltraEdit: Instruction-based Fine-Grained Image Editing at Scale.\" arXiv preprint arXiv:2407.05282 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging that our approach is innovative, \"EvolveDirector offers a feasible solution to access advanced text-to-image generation capabilities in an open-source environment\", \"The trained model, Edgen, shows impressive performance\", \"The paper includes extensive experiments and ablation studies \",  and \"The methodology is technically sound, with a clear explanation ...\", etc.\n\nWe will address your concerns as follows.\n\n----\n\n**W.1**: \nIn the revision, we will consider shrinking the space occupied by Figure 1, and use the saved space to add more discussions and analyses.\n\n**W.2**: \nThank you for acknowledging the two research problems. Generating large-scale data based on the naive pipelines to train the base model is extremely costly and time-consuming (hundreds of A100 GPU days for training the base model on million-scale data), let alone training multiple base models to approach multiple advanced models. To tackle this question more efficiently, we simplify the task by fixing the same base model and advanced model to examine the impact of different data scales on the performance of training the base model to approach the advanced model, as shown in the 7 sets of experimental results in Table 2. The results show that the generative capabilities of the advanced model can be learned through training on its large-scale generated data, which is over 11 million. However, if the training data is reduced to 10% or even 1% of the original amount, the performance of the trained model will significantly decrease. The proposed EvolveDirector can train the model in a data-efficient manner, thus offsetting the decrease in model performance caused by reducing the amount of training data.\n\nRegarding question 2, the base model can gain better performance by learning to approach multiple advanced models simultaneously, thus providing an answer to this question. This success is attributed to VLM selecting the best ones from generated images of multiple advanced models as training data.\n\nWe will add the discussions and clarification in the revision. \n\n**W.3**: \nThanks for your comment. Introducing real images as base images and modifying them to generate more training samples is an interesting idea. The mentioned work [1] is suitable to benefit the instruction-based image editing task. However, when pre-train a T2I foundation model, this pipeline may struggle to generate imaginative samples since it creates new samples based on editing the real ones. Besides, in the paper, we focus on exploring how to approach the advanced models based on their generated data. The experimental results demonstrate that relying solely on generated data is sufficient to approach advanced models.\n\nWe will add an introduction to this reference [1] in the related works. \n\n**W.4**: \nPlease note that the data volumes of these two training methods are significantly different, as shown in Table 2. With Discrimination & Expansion & Mutation, the base model can achieve similar generation performance on a much smaller data scale. We will highlight this to avoid the confusion.\n\n----\n[1] Zhao, Haozhe, et al. \"UltraEdit: Instruction-based Fine-Grained Image Editing at Scale.\" arXiv preprint arXiv:2407.05282 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our gratitude to the reviewers for their insightful comments. We appreciate the recognition of the strengths of our paper, including the novelty and effectiveness of the proposed EvolveDirector, and the recognition of extensive experiments and ablation studies. 

We would like to highlight the following recognitions from the reviewers：

- Reviewer gzrb: "This opens up an avenue of research into better use of expert models for informing the training procedure in T2I."

- Reviewer i5a7: "The proposed framework is novel" and "The way of utilizing VLMs is interesting and effective."

- Reviewer 9FQn: 
"EvolveDirector offers a feasible solution to access advanced text-to-image generation capabilities in an open-source environment", "The trained model, Edgen, shows impressive performance", and "The paper includes extensive experiments and ablation studies that validate the framework's effectiveness."

We address the concerns of reviewers in detail in the subsequent responses respectively.

Meta-review: The work uses vision-language models to augment text-to-image generation. The combination of these two kinds of models is interesting. All reviewers agree that the paper can be accepted. The author responses solved most questions, which can be included in the final version.
Decision: Accept (poster)