EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models

Authors: rui  zhao
hangjie  yuan
yujie  wei
shiwei  zhang
yuchao  gu
lingmin  ran
xiang  wang
zhangjie  wu
junhao  zhang
yingya  zhang
mike zheng shou
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-28 18:26:42

Abstract: Recent advancements in generation models have showcased remarkable capabilities in generating fantastic content. However, most of them are trained on proprietary high-quality data, and some models withhold their parameters and only provide accessible application programming interfaces (APIs), limiting their benefits for downstream tasks. To explore the feasibility of training a text-to-image generation model comparable to advanced models using publicly available resources, we introduce EvolveDirector. This framework interacts with advanced models through their public APIs to obtain text-image data pairs to train a base model. Our experiments with extensive data indicate that the model trained on generated data of the advanced model can approximate its generation capability. However, it requires large-scale samples of 10 million or more. This incurs significant expenses in time, computational resources, and especially the costs associated with calling fee-based APIs. To address this problem, we leverage pre-trained large vision-language models (VLMs) to guide the evolution of the base model. VLM continuously evaluates the base model during training and dynamically updates and refines the training dataset by the discrimination, expansion, deletion, and mutation operations. Experimental results show that this paradigm significantly reduces the required data volume. Furthermore, when approaching multiple advanced models, EvolveDirector can select the best samples generated by them to learn powerful and balanced abilities. The final trained model Edgen is demonstrated to outperform these advanced models. The framework EvolveDiretor and the trained model Edgen will be fully open-sourced to benefit the downstream tasks.

Main paper: 
 introduction: 
 In the field of AI-generated content, an increasing number of advanced models have showcased their ability to generate realistic and imaginative images, such as Imagen [1] , DALL•E 3 [2] , Stable Diffusion 3 [3] , Midjourney [4] . While these models benefit from publicly available datasets such as ImageNet [5] , LAION [6] , and SAM [7] , they rely more heavily on proprietary, high-quality data collections that surpass the quality of publicly accessible datasets. Models such as Midjourney [4] are particularly noted for deriving substantial benefits from their internal datasets. However, given the significant commercial advantages brought by their impressive capabilities, most advanced models keep their parameters private, hindering reproducibility and democratization. In this paper, we aim to explore training an open-source text-to-image model with public resources to achieve comparable capabilities with the existing advanced models.Three transparent glass bottles on a wooden table. The one on the left has red liquid and the number 1. The one in the middle has blue liquid and the number 2. The one on the right has green liquid and the number 3.At the foot of the snow-capped mountain, a clear stream winds its way, murmuring and flowing gently, resembling a silver ribbon that connects the forests and canyons.A volcanic eruption in a dense jungle creates a striking scene as lava flows down the mountain, contrasting with the lush green foliage. Dark clouds fill the sky, casting dramatic shadows over the landscape.A sleek and modern logo design for NeurIPS 2024. The text "NeurIPS 2024" is written in bold, black capital letters. In the background, there's a minimalist, abstract representation of a brain, with a few key parts highlighted in neon colors. The overall design is clean, visually appealing, and instantly recognizable.A captivating book cover features a young woman with an enchanting gaze, dressed in vibrant attire, surrounded by swirling colors, inviting readers into a world of chaos and beauty.Pallet knife technique, large intentional brushstrokes, masterpiece painting of a leopard silently prowling through the underbrush, dark fantasy.In a garden, there sits a white cat on the left and a black dog on the right, with a green water cup placed between the two of them.Four superheroes stand in front of the pyramids, from left to right, Batman, Hulk, Iron Man, and Spider-Man.Figure 1 : Images generated by our model Edgen (EvolveDirector-Gen). Edgen can generate highquality images with multiple ratios and resolutions. Notably, it excels in generating text and avoiding attribute confusion when generating multiple objects, which are significant characteristics of the most advanced text-to-image models available today. The input text prompts are annotated under the corresponding images.Despite the fact that internal data and model parameters of many advanced models remain inaccessible, they provide publicly accessible application programming interfaces (APIs) that enable users to access their generated distribution. This leads to the construction of synthetic benchmarks, e.g., JourneyDB [8] collects 4.7 million images generated by Midjourney [4] . This benchmark is further utilized for enhancing the training of new generative models [9] . However, this paradigm is not data efficient, posing challenges in terms of substantial computation and expenses. Instead of statically constructing multiple expensive large-scale datasets for each advanced model, we take a step forward in this paper by delving into recovering their generative capabilities in a unified framework with limited samples. We propose EvolveDirector to address this challenging task by shedding light on two research questions: (1) How many synthetic text-image pairs are sufficient to approximate the generative capability of an advanced model? (2) Taking it a step further, is it possible for the base model to obtain generative capabilities beyond the advanced models?To explore the first question, we start with a demonstration experiment by training a relatively poor model, a DiT model [10] pre-trained on public dataset ImageNet [5] and SAM [7] , to approach the advanced model PixArt-α [9] using increasing data scales. The training data is curated by collecting diverse text prompts and utilizing them to generate images from PixArt-α. The experiments indicate that when we scale the training data (i.e., generated image-text pairs) to 11 million, we can obtain a base model achieving similar capabilities to the target model without access to its internal data. However, the magnitude of 11 million generated data is comparable to the 14 million data used for pre-training the target model. Training a base model in this way incurs significant expenses, not only in terms of time and computational resources but also the costs associated with using fee-based APIs of some advanced models.For more efficient training, it is crucial to minimize data redundancy and maximize data quality, as the marginal benefit of training on inferior data is limited. The corpus of the 11 million text prompts, generated from the SAM dataset [9] , and the images generated by the advanced model exhibit redundancy in several aspects: (1) lacking imaginative text prompts due to the photographic nature of SAM images; (2) high similarities among text prompts; and (3) imbalanced data quality. The generated images using the target advanced model may exhibit low quality due to inferior alignment on some text prompts. To address these problems, we introduce large vision-language models (VLMs) to improve the diversity and quality of training data for efficient training. Our approach involves a continuous evaluation with VLMs to dynamically refine the training dataset by the discrimination, expansion, deletion, and mutation operations. This dynamic curation strategy ensures that only valuable data is retained, significantly reducing the volume of data for training. Experimental results demonstrate that a mere 100k training samples are sufficient for the base model to gain similar performance to that of the target model, which is substantially fewer than the 11 million samples required by the baseline method.To explore the second question, we applied EvolveDirector to train the base model to approach multiple recent most advanced models in a unified framework, including DeepFloyd IF [11] , Playground 2.5 [12] , Stable Diffusion 3 [3] , and Ideogram [13] . For each text prompt, we invoke advanced models to generate their images, and the VLM selects the best match to train our base model. The final trained model is named Edgen (EvolveDirector-Gen). The experimental results demonstrate that Edgen outperforms the advanced models mentioned above. Although our initial goal was to approximate these advanced models, we ultimately benefited from the VLM in choosing better training samples from their generated data, thereby achieving capabilities superior to any individual model.The code of EvolveDirector and the model weights of Edgen are released to benefit the downstream tasks. Our contributions are summarized as: (1) Through experiments on massive data, we conclude that the generation abilities of a text-to-image model can be approximated through training on its generated data. (2) We propose the EvolveDirector, a framework that harnesses VLM to direct the training of the base model to learn generation ability from advanced models efficiently. (3) The trained text-to-image model Edgen outperforms the most advanced models. 
related works: 
 Text-to-Image Generation. To advance the overall quality of text-to-image generation, research efforts have been invested in exploring architectural improvement [10, 14, 15, 16] and generation paradigm advancement [17, 18, 3] , etc. Diffusion models stand out as the de facto text-to-image generation paradigm [19, 20, 1, 3, 21, 22, 23, 24, 25] , noted for its scalability and stability [26] . They benefit numerous downstream tasks, spanning image editing [27, 28] , video generation [29, 30, 31, 32] , 3D content generation [33, 34] , etc. Through the use of highly descriptive and aligned image-text pairs at a substantial scale, text-to-image models that excel in resolutions, safety control, and the capability to render accurate scenes are obtained, e.g., Imagen [1] , Midjourney [4] , DALL•E 3 [2] , Stable Diffusion 3 [3] , and Ideogram [13] . However, the exceptional capabilities of most advanced models have led providers to restrict access, typically offering only APIs, which limits their widespread and equitable use. In this paper, we aim to fill this gap by leveraging advanced VLMs to direct base model to replicate the functionality of advanced models. In contrast, some works propose to motivate models to learn from their self-generated images [35, 36] .Evaluating T2I Generation with VLMs. Some automatic evaluation methods [37, 38, 39] are propoesd to combine the LLMs and VQA models to evaluate the generated contents. Thanks to the substantial advancement of large language models [40, 41, 42, 43, 44, 45] , the capabilities of VLMs are largely boosted [46, 47, 48, 49, 50, 51] . Utilizing these enhanced capabilities, methods building on VLMs are designed to facilitate the evaluation of text-to-image generation [52, 53, 54, 37, 55] . Notably, the recent research effort, Gecko [54] , demonstrates the practicality of leveraging pretrained LLMs [44] and VLMs [45] for systematic evaluation of text-to-image generative performance, spanning aspects such as text rendering, relational generation, attribute generation, etc. However, previous research requires fine-grained evaluation across various aspects, which remains challenging. In EvolveDirector, we simplify this process by requiring only pairwise comparisons, which enables a stable and reliable performance, facilitating the approaching of advanced text-to-image models.Knowledge Distillation. KD [56] aims to transfer knowledge from well-trained teacher models to a simpler student model. The primary focus of most research in KD lies in investigating distillation losses with the output predictions of teacher model [57, 58] , intermediate feature maps [59, 60] , or feature correlations [61, 62] to distill knowledge. Recently, distillation methods based on diffusion models have garnered attention, primarily by distilling outputs from intermediate steps of the diffusion process to expedite the sampling process [63, 64, 65, 66] . Despite sharing the same goal of approaching the performance of advanced models, we would like to highlight our training paradigm is an orthogonal procedure to KD. To approach the advanced models with only APIs available, we avoid the need for distillation losses or acquiring intermediate results and instead choose a data-driven paradigm. Furthermore, our designed paradigm is also orthogonal to dataset distillation [67] as we evaluate and refine data during training rather than relying on a pre-existing large dataset.Online Learning. Online learning has garnered significant interest for its capacity to enable models to adapt to real-time and dynamic data scenarios [68, 69] . This attention extends to a variety of real-world tasks, including semi-supervised learning [70, 71] , unsupervised learning [72, 73] , and continual learning [74, 75, 76] , etc. In EvolveDirector, we harness the potential of online learning and powerful VLMs to evolve models towards advanced generation models, offering an efficient and scalable framework capable of dynamically adapting to evolving data. 
method: 
 In this section, we outline the EvolveDirector framework in Sec. 3.1, describe the detailed operations of the VLM within this framework in Sec. 3.2, and discuss the training strategies in Sec. 3.3.The proposed framework EvolveDirector, as shown in Fig. 2 Interaction with Advanced Models. While the model configurations and weights of many advanced T2I models are not publicly accessible, they often provide APIs for interactions. In EvolveDirector, we interact with these APIs by submitting text prompts and receiving the corresponding generated images. The one that aligns with the given text prompt better will be selected by the VLM and Training of the Base Model. Based on the dynamic training set, we train a Diffusion Transformer (DiT) [10] as our base text-to-image model. Specifically, the model is built upon the improved architecture proposed by PixArt-α [9] . Besides, we incorporate layer normalization after the Q (query) and K (key) projections in the multi-head cross attention blocks [77] to further stabilize the training,Attention(Q, K, V ) = softmax f Q (Q)•f K (K) T √ d kV , where f Q (•) and f K (•) are the layer normalizations after the projections.The VLM acts as a director to guide the construction of more valuable dynamic datasets. To simplify the task difficulty and better leverage the capabilities of VLM, we present it with a choice of two images as a multiple-choice question, as shown in the top left corner of Fig. 3 . One image I advanced is generated by the advanced model, while another one I base is generated by the base model, respectively. In practice, the order of the two images is randomized. VLM is called to compare them and decide which one aligns better with the given text prompt T . Regarding the choice of VLM, there are two potential scenarios as follows.(1) I advanced outperforms I base . The inferior performance on this text prompt indicates that the base model is still under-trained. Therefore, EvolveDirector utilizes the VLM to generate more N S variations of this text prompt T , as shown in the lower-left corner of Fig. 3 . Then the advanced T2I model generates corresponding images to expand the dynamic training set, as shown in the right side of Fig. 3 . The original samples will continuously be involved in the training.A medieval knight riding a majestic steed galloping down a cobblestone street in a castle town.A post-apocalyptic survivor on a rugged motorcycle traversing a desolate wasteland.A futuristic warrior on a hoverbike navigating through a neon-lit cityscape.A steampunk inventor on a mechanical velocipede exploring a clockwork-themed industrial landscape.A fantasy elf on a magical unicorn ascending a mystical waterfall in a forest realm.In these two images (A) and (B), which one aligns better with the text description "Ancient soldier riding a horse approaching along a winding mountain path from afar."? You have two options: <(A) is better>, <(B) is better>.Give me your detailed explanation of your comparison and analysis.Here's my detailed analysis: Both images depict a figure on horseback, but the armor and attire of the rider in (A) are more consistent with ancient soldier attire. The rider in (A) is wearing a helmet and armor that resemble historical Roman or Greek soldier gear, which is more in line with the description of an "ancient soldier."Replace the nouns in the text description with other kinds of objects, characters, backgrounds, names, colors, or styles to generate 5 more diverse text descriptions. Besides the expansion and deletion operations, EvolveDirector also performs mutation operations with a certain probability. This operation permits the VLM to generate more diverse text prompts independent of any existing ones, thereby encouraging the model to explore and learn from a broader domain of text prompts. Stable Training. In our task setup, the scale of training data is significantly less than the million scale. Under this circumstance, the original architecture [9] demonstrates considerable instability during training, and the generation collapse is observed. Thus we follow the work [77] and apply the layer normalization after the query and key projections to improve the training stability. The experimental results detailed in the supplementary demonstrate the effectiveness of this adaptation.Multi-Scale Training. The ability to generate images across various scales and aspect ratios is a significant capability of advanced T2I models. To facilitate more efficient training, we initially train the base model on images with a fixed resolution of 512px. Subsequently, we extend the training to images of higher resolution with multiple aspect ratios, thereby enabling the model to generate multi-scale and multi-ratio images. Following [9] , we construct buckets with different aspect ratios and image sizes. In EvolveDirector, for each text prompt, a size bucket is randomly sampled from the buckets and advanced T2I models are called to generate images with this size. To avoid generation collapse, if the selected bucket falls outside the optimal size range of the advanced model, it will be resized to the closest appropriate size. 
training details: 
 We train the base model on 16 A100 GPUs for 240 GPU days, with a batch size of 128 and 32 for images at 512px and 1024px resolution, respectively. The VLM evaluation process is distributed across 8 A100 GPUs to facilitate its speed. The open-source advanced models are deployed on our devices with simulated APIs for interaction. For closed-source models, EvolveDirector interacts with them through their public APIs. The ability of the VLM to analyze images is crucial to the EvolveDirector framework. There are multiple powerful VLMs available, including CogVLM [82] , CogAgent [78] , Qwen-VL (Qwen-VL-Chat, Qwen-VL-Plus, and Qwen-VL-Max) [42] , InternVL [79] , LLaVA and LLaVA-Next [51, 80] , and GPT-4V [81] . We evaluate their newest version on 600 pairs of questions to calculate the alignments with human raters. Each question consisted of a text prompt and two images generated based on that prompt, presented to 5 different human raters. The VLMs are tested in two aspects: (1) Discrimination, and (2) Expansion. To be more specific, (1) initially, the VLMs were required to select which image aligned more closely with the given text prompt. The output is scored by 0 for wrong or 1 for correct by human raters. (2) Subsequently, they are instructed to generate more variations of the given text prompt. The score of "Accuracy" evaluates whether the generated text prompts contain any linguistic errors and whether they are in the same syntactic structure as the given text prompt. The score of "Diversity" evaluates the diversity among the generated text prompts. These two scores range from 1 (worst) to 5 (best). The results are shown in Tab. 1. The LLaVA-Next and GPT-4V achieve the top performance. Considering that LLaVA-Next is totally free to use, we select it as the VLM for EvolveDirector. Models. For ablation studies, we select the Pixart-α [9] as the unified advanced model to approach. We utilize a DiT model pre-trained on publicly accessible data, i.e. the ImageNet and SAM dataset, as the base model.Metrics. We sample 10, 000 text prompts to feed into models trained under different ablation settings to generate images and calculate their FIDs with the images generated by the advanced model. These text prompts are not seen by these models in the training stages. To conduct human evaluation, we randomly selected 300 sets of text prompts and their corresponding generated images, which are 2, 400 in total. These images are paired in twos, each consisting of one image from the advanced model and one from an ablation model corresponding to the same text prompt, arranged in random order. Then they are presented to 5 different human raters to choose which image matches the text prompt better. In this manner, each ablation model is paired with the advanced model for comparison, and their selected ratios are recorded to reflect their relative performance compared to the advanced model.Results. The results of FIDs and human evaluation are shown in Tab. It is worth noting that there is still a slight gap between the final model and the advanced model, which can be attributed to the law of diminishing returns. This occurs because it becomes increasingly difficult to identify truly high-value samples as the performance approaches that of the advanced model. However, this phenomenon vanishes when EvolveDirector learns from multiple advanced models simultaneously, as experiments in Sec. 4.4 demonstrated. This is because the larger performance gaps between multiple models facilitate the easier identification of high-value samples. We select several latest advanced models to approach their powerful generation ability, including Playground 2.5 [12] , Stable Diffusion 3 [3] , and Ideogram [13] . Playground 2.5 is famous for its aesthetic generative effects, while Stable Diffusion 3 and Ideogram are known for their strong performance in various aspects including text generation and multi-object generation. Besides, we select a relatively old model, DeepFloyd IF (but just released in April 2023) [11] , for its amazing text generation ability. The base model is the same as the one in Sec. 4.3, and we continue to train the base model that has already approached the Pixart-α to approach the selected multiple advanced models. During training, EvloveDiretcor will feed each text prompt to all of them to generate corresponding images, and then use VLM to select the best one of them as the image from the advanced model. The selected image then undergoes evaluation as detailed in Sec.3.2.Select Ratios of Advanced Models. We have calculated the proportions of images generated by these models that were selected by the VLM in the training stage, as shown in Table 3 . Among the generated images, those generated by Ideogram are selected with the highest ratio. Besides, we evaluate the select ratios of images in specific domains, such as human generation, text generation, and multiple object generation. Examples of these three types are shown in Fig. 5 . The results in Table 3 show that different advanced models achieve the highest select ratios in different specifics. For example, for generating text in images, the Stable Diffusion 3 outperforms others, while the Playground 2.5 is much worse than others. This may be caused by that the internal training data of Playground 2.5 does not include sufficient high-quality images with text in them. This also demonstrates the significance of diverse high-quality data.Quantitative Comparison. To evaluate the performance of the trained Edgen, the base model, and the advanced models, we sample text prompts and feed them into each model to generate images. One image generated by Edgen and one image generated by the comparison model are combined together. Finally, same with the scale of comparison in previous works [9] , 300 text prompts and 1800 image combinations are shown to human raters to select which one aligns with the given text prompt better. Each combination is evaluated by 5 different human raters. The results are shown in "A seasoned journalist, his charismatic face framed by shelves of archived stories and reports. His eyes, sharp and scrutinous, glance over the rim of his glasses at the historic headlines that surround him." "A cheerful mug on a cluttered office desk, its side emblazoned with 'Make Today Amazing' in vibrant, uplifting colors." "A small wooden boat drifting on a calm lake, containing a young child wearing a life jacket, a playful puppy with its tongue out, and a calm old cat dozing off beside them." the images generated by multiple advanced models to be used as training data. Therefore, although EvolveDirector initially aims to train the base model to approach them, the final trained model Edgen outperforms all of the advanced models. Besides, we evaluate the performance of these models on specific types of text prompts, with the results displayed on the right side of Fig. 4 .Qualitative Comparison. In Fig. 5 , we showcased three groups of generated images. The three rows of results respectively demonstrate the generation capabilities for humans, text, and multiple objects. For the first group, only the images generated by Edgen, DeepFloyd IF, and Ideogram successfully reflect the "face framed by shelves of ...". As shown in the second row, only Edgen, Ideogram, and Stable Diffusion 3 have the ability to generate correct text in images. For the results shown in the third row, three objects need to be generated, i.e. the child, puppy, and cat. Only Edgen and Ideogram success and other models lost some objects. These results show that Edgen has already learned powerful generation abilities and outperforms some advanced models. 
conclusion: 
 In this paper, we propose EvolveDirector, a framework that targets approaching the generation capabilities of advanced text-to-image models by only utilizing their publicly accessible APIs. By harnessing the capabilities of large vision-language models for evaluating image-text alignment, EvolveDirector significantly reduces the volume of training data required, thus saving considerable training costs, especially those associated with API usage. Experimental results demonstrate that the resultant model Edgen, inheriting the generation capabilities from multiple advanced models, achieves superior performance in various aspects. The limitations and future work are discussed in the supplementary. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We claim social impact in the appendix Sec. B.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The safeguards are checked by the advanced models internally.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The code, data, and models are all properly credited and their license and terms of use are properly respected in our work.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: While EvolveDirector achieves significant strides in approximating the generation capabilities of advanced text-to-image models, the resultant model can face challenges related to bias. Our model can inadvertently inherit biases present in the images generated by the advanced models. Additionally, our reliance on VLMs to evaluate generated images could introduce their own biases into the selection process. To mitigate these issues, it is better to integrate debiasing methods and incorporate human feedback in future developments, aiming to enhance the robustness and fairness of EvolveDirector.Our research leverages the foundational capabilities of large vision-language models to reproduce advanced text-to-image models. This initiative represents a pioneering approach to make state-ofthe-art generative technologies more accessible and cost-effective. Our methodology substantially reduces the volume of data required for model training, thereby reducing computational demands and minimizing environmental impacts associated with image generation. The resultant model Edgen has the potential to revolutionize the creation of digital media owing to its inheritance of capabilities from multiple advanced models. For instance, its capabilities enable the creation of diverse media content and marketing materials, such as logos, by effectively rendering text into realistic and imaginative visual representations.However, we also acknowledge potential negative societal impacts. As with many generative technologies, there is an inherent risk of bias in terms of gender and race in the generated content. Furthermore, the improper use of the resultant model, such as inputting harmful or obscene text prompts or creating unauthorized or deceptive images of public figures, could facilitate misinformation or impersonation. To mitigate these risks, it is imperative to implement thorough oversight to secure its ethical use. after QK projections a 3d render of a cruise ship in a modification garage, side perspective of cruise ship, engine placed outside, engineers fixing the cruise ship. The background features a ship garage with high lights, daylight and ship spare parts. the images exudes captivating cinematic expression, 4k, 3d render a man standing on a pier at the beach, looking out at the ocean. The scene is set during sunset, with the sun casting a warm glow on the sky and the water. The man is wearing a hat and a black shirt, and he is holding a camera, possibly taking a photograph of the beautiful sunset. The combination of the man's attire, the setting, and the monochromatic color scheme creates a visually appealing and evocative image. Instruction for Discrimination. We input the combination of two images and instruct the VLM to choose which one aligns with the given text prompt better. The instruction is as follows, "In these two images (A) and (B), which one aligns better with the text description "text prompt"? You have two options: <(A) is better>, <(B) is better>. Simply state your choice, no need for a detailed explanation." The enhanced prompt is randomly sampled from the following ones:Instruction• "It should be rough and short."• "It should contain less than 30 words and be highly detailed."• "It should contain over 30 words with different granular levels of detail."• "It should contain over 50 words with a lot of details."Structure the Outputs of VLM. The diversity of output formats from VLM can pose challenges for automated parsing. We found that by providing specific instructions to the VLM, its output format can be standardized. Specifically, when prompting the VLM to generate more text prompts, we offer instructions such as "Arrange them in the format of a list ["Text description 1", "Text description 2", ...]." This approach directs the VLM to generate outputs in a consistent format. The model is trained by the AdamW optimizer with a learning rate of 2e-5 and with a gradient clip of 1.0. A constant learning rate schedule with 1000 warm-up steps is used. The gradient checkpointing is used to save the VRAM. For generating 512px images, we train the base model with 100K training steps, and for 1024px image generation, we train the model with 20K training steps.The default number of training samples at the beginning of training is 20K, which will gradually grow to the upper bound. For the ablation studies in Sec. 4.3, we set the upper bound to 100K to explore the extreme performance of EvolveDirector. The initial text prompts are sampled from those captioned for the SAM dataset [9] . For approaching multiple advanced models in the Sec.4.4, we setThe words 'space' are inscribed at the top in bold, cursive letters, with a effect that matches the vibrant colors of the scene. A stunning and intricate 3D render of an astronaut sitting positionamidst a miniature galaxy. The astronaut is positioned sitting in front of a vibrant, detail-rich landscape of big all planets, each with its unique color and texture. An extensive jasmine flower garden surrounds the astronaut , adding an element of realism and beauty to the scene. The cinematic and conceptual art style of the illustration captures the essence of a cosmic adventure, evoking a sense of wonder and exploration., portrait photography, conceptual art, photo, illustration, 3d render, cinematic, vibrant.WEARS A GLASES WITH BACK GROUND GREEN, 3d render A detailed and vibrant portrait of a young female character. She has striking blue eyes, purple hair with two buns on top, and her hair flows down in braided pigtails. Her skin glass like skin and she has a light pink lip color.The background is minimalistic, with splashes of color that seem to mimic paint splatters. The character is wearing a ruffled off-shoulder top in shades of purple and pink., painting, vibrant.A breathtaking, serene scene of a graceful iridescent swan gliding effortlessly across a tranquil body of water. The swan's pristine feathers contrast beautifully with the warm and soothing colors of the sky, which are painted in shades of blue, and gold. The water mirrors the sky's hues, creating a stunning mirror image of the swan and the enchanting sky. The swan's elegantly curved neck and slightly open beak, revealing a hint of its vibrant orangered interior, add a touch of delicate beauty to the scene. The overall atmosphere of this captivating image is one of peace, tranquility, and breathtaking natural beauty., photo, cinematic, wildlife photography, dark fantasy, anime.A cartoon panda riding a skateboard with a pumpkin. A stunning, close-up photograph of a delicate, vibrant flower with an intricately detailed butterfly resting gently upon its petals. The flower, with its vivid colors and enchanting shape, is the centerpiece of the image. The butterfly, with its strikingly beautiful patterns and wings spread wide, contrasts against the flower's rich hues. The background is a serene blur of soft greens, creating an overall sense of tranquility and harmony in nature.A vibrant and eye-catching 3D illustration of an orange heart-shaped slice, bursting with color and energy. The slice is surrounded by a riot of different colors, with a vivid sunset-like sky in the background. The overall effect is playful and joyful, with a sense of whimsy and delight, as if the orange slice is a symbol of love and happiness., illustration, 3d render.A heart-shaped orange slice, colorful background, illustration, 3d render.A portrait of a man which is looking excitedly forward to learning about and trying artificial intelligence tools., illustration, photo. the upper bound to 300K to ensure a more sufficient learning of much powerful generation abilities of the most advanced models. The initial text prompts are randomly selected from both the SAM captioning dataset and the community-sourced text prompts. A colorful comic-style linocut illustration of a captivating pop art-inspired composition, a female model walking on a pink carpeted runway. She is wearing a unique, offshoulder dress with a draped design that exposes parts of her body. The dress is in a light blue color and has a sheer, transparent section at the bottom. The model has a short, blonde hairstyle and is looking directly at the camera. The background is a mesmerizing blend of multi-colored abstract shapes and mandala patterns, creating a harmonious and visually striking scene.The head of a lion with majesty and elegance. Chin high and pride. Her hair is long and voluminous, and at its ends you can see trails of light and fire coming out of her hair.In the background you can see the Savannah with a few animals in the distance such as giraffes or birds.Illustrate a delicate scene where a small, intricately detailed hummingbird with a speckled blue-green throat hovers near a cluster of large, soft pink flowers. The setting is dreamlike, with a misty, orange-toned background suggesting either dawn or dusk. Focus on the light reflecting off the bird's iridescent feathers and the gentle sway of the flower petals., painting.A charming and captivating illustration of a Maltese puppy wearing a colorful, enchanting witch's hat, attend a glamorous garden party under the moonlight. The puppy is smiling and enjoying the warm atmosphere, surrounded by vibrant, glowing flowers. The background is filled with colorful bokeh, and the scene is bathed in a soft glow. Sharp details and a shallow depth of field create a sense of depth and dimension. The overall atmosphere is cozy and cinematic, with a touch of fairy tale enchantment.An amusing and whimsical image of four multicolored gummy bears gathered under a small umbrella. A large spoon hovers above them, pouring a shower of sugar over the gummy bears, creating a sweet and playful scene. The background is a bright and colorful dreamscape, filled with an assortment of candies, pastel clouds, and delightful surprises.A charming illustration of a dog with its mouth open, showing off its playful and friendly personality. The dog has thick fur and a mischievous glint in its eyes, with a wagging tail that exudes joy. The background features a cozy home with a warm fireplace, a sunny yard, and a colorful garden., illustration.A breathtaking photo-realistic image of a confident and alluring old classy gentleman, striding down a bustling city street. Dressed in an elegant XIX century suit."A captivating, cinematic photorealistic image of a group of people standing on a dramatic cliff, their silhouettes accentuated by the soft glow of the full moon. The ocean below is a raging sea with towering waves, creating a sense of awe and danger. The sky is a rich blend of deep blues and purples, with the moon casting a shimmering reflection on the water. The overall atmosphere is eerie yet mysterious, as if the group is witnessing a rare lunar event., photo, cinematic. A stunning beach art print capturing a serene coastal scene.In the foreground, a meticulously painted seagull is perched on a rock, its wings outstretched and eyes fixed on the horizon. The golden sand stretches out, dotted with footprints that seem to disappear into the distance. A vibrant sun is setting on the right, casting a warm, golden glow on the waves that gently lap at the shore., painting.A serene and romantic image of a young woman with long brown hair and glasses, seated on a striped beach chair. She wears a yellow off-shoulder dress that complements her calm demeanor. The starry night sky above her is dotted with countless twinkling stars, and a distant cityscape adds a soft glow to the scene. The reflection of city lights on the water creates a tranquil and intimate atmosphere, perfect for contemplation.A meticulously carved wooden relief depicting a little cute greydoodle puppie and a cute beagle.A cityscape built on the back of a giant turtle. Justification: We provide information on the computer resources in Sec. 4.1Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The work uses vision-language models to augment text-to-image generation. The combination of these two kinds of models is interesting. All reviewers agree that the paper can be accepted. The author responses solved most questions, which can be included in the final version.

Reviews: review: paper_summary: paper_summary: This paper explores the effectiveness of training a text-to-image (T2I) model using synthetic examples generated by existing T2I models. The authors find that on the order of 10M image-text pairs are necessary to approach the quality of a good model like PixArt-Alpha, while using only 1M or 100k examples results in serious deterioration.

To improve sample efficiency, the authors introduce their online learning framework, EvolveDirector, which leverages a vision-language model (VLM) to curate a better online training set. The VLM is used to compare the current trained checkpoint's generated images against the reference T2I model's images, in order to determine which training examples to drop from the training set, and which training example prompts to generate more variants of for further training. Occasionally, entirely new training prompts are introduced to increase diversity. Overall, this results in a training process which matches the quality of PixArt-Alpha according to human comparisons using only 100k examples.

The authors perform further ablations to show the necessity of each part of the framework, and train a model using four different advanced T2I models as targets, resulting in a final T2I model, named Edgen, which surpasses each of the target models in overall human preferences.
strength_weakness: strength_weakness: **Quality.** While most of the experimental details are explained thoroughly, the paper omits any description of the text data used. The text data is a crucial part of the proposed framework, since it is used to seed the training data, as well as for the mutation operation, presumably. There is also no mention of the source of the data used for evaluation. Along the lines of evaluation, the dataset used to compute FID score should also be mentioned.

**Clarity.** The paper fails to clarify many useful details earlier in the paper (abstract + introduction), leaving the overall understanding of the method vague until reaching Section 3. For example, the key components of expansion, deletion, and mutation are mentioned but not explained at all in the introduction.
questions: questions: 1. What is the source of your training data? The training images are generated by the expert/advanced T2I models, but where do the input prompts come from?
2. Following Q1, how do you know the seed prompts for training (i.e. the initial training set) cover a diversity of prompts? While using LLMs to create variations of prompts can result in new combinations of objects or concepts, the resulting new prompts are unlikely to capture the same diversity of text present in natural datasets.
3. Where does the *mutation* operation source the new prompts from? Are they also generated by the same VLM, or taken from the same source as your initial training data? If they come from an existing dataset, what is the benefit of adding them midway through the training process instead of just starting with a initial training set?
4. To my understanding, LLaVA-NeXT (and many other current VLMs) are not trained on examples involving multiple images. Are the images for comparison concatenated or passed separately to LLaVA? Do you notice any statistical confounders, such as choosing the first image more often than the second?
5. The instructions for "expansion" to the VLM, to generate new variations on the prompt, specify only to replace nouns in the prompt. Why is this the strategy used to generate similar prompts? What happens if the T2I model is struggling to generate a specific noun, and the syntactic structure of the prompt is not the weak point? In that case, it feels like the new prompts would not be relevant for further training.
6. When selecting which VLM to use, you say "The output [of the VLM] is scored by 0 for wrong or 1 for correct by human raters". Why not ask the human raters to rank the generations themselves, and then compute how many examples the VLM and the humans agreed on? Having the human raters rate the VLM output seems like it could induce bias in the raters.
limitations: limitations: The authors note that their method could result in a T2I model which inherits biases from both the teacher T2I model and the VLM used for evaluation. They also describe the potential positive and negative broader societal impacts of the research, including improving model training costs and potential for misuse.
paper_summary: paper_summary: This paper investigates how to train the text-to-image model comparable to advanced models using publicly available resources. Specifically, EvolveDirector collects training data with the APIs of advanced models, and further uses a VLM to continuously refine the training dataset. The proposed VLM refinement significantly reduces the data volume needed to teach a base T2I model and improves the training efficiency.
strength_weakness: strength_weakness: 1. The explanation of how to instruct VLM is not detailed. For example, how to ensure the generated text prompts can be parsed correctly and free of errors, which could introduce noise into the training data.

2. It would be better to also evaluate the final model on other benchmarks (e.g., DSG, TIFA).

3. Missing references: 
(1) DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback
(2) SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
questions: questions: Typos: in line 336, “outperform” -> “outperforms”.
limitations: limitations: N/A
paper_summary: paper_summary: The paper introduces "EvolveDirector," a novel framework designed to train a text-to-image generation model that can compete with advanced models using only publicly available resources. The authors aim to address the limitations posed by proprietary data and the inaccessibility of parameters in state-of-the-art models by leveraging public APIs to obtain text-image pairs for training.

The core of EvolveDirector relies on the interaction with advanced models through their APIs to generate data for training a base model. A significant challenge is the need for a large volume of data (10 million samples or more), which is both time-consuming and computationally expensive. To mitigate this, the authors employ pre-trained large vision-language models (VLMs) to dynamically update and refine the training dataset through various operations like discrimination, expansion, deletion, and mutation.

The framework's efficiency is demonstrated through experiments that show a significant reduction in the required data volume, with only 100k samples needed for comparable performance to models trained on 11 million samples. Furthermore, the paper claims that their final trained model, "Edgen," outperforms several advanced models and will be open-sourced for downstream tasks.

The paper is well-written and presents a strong case for the capabilities of EvolveDirector. It would be beneficial to see additional experiments or comparisons with other state-of-the-art models to further establish the framework's superiority. Moreover, the paper could delve deeper into the specific mechanisms of knowledge transfer from the VLMs to the base model.
strength_weakness: strength_weakness: 1. The layout needs to be adjusted. For example, Figure 1 takes up an entire page.
2. The research question 1 & 2 is insightful, which is an important research topic in the perspective of distillation and data annotation.. However I see limited analysis or exploration for this question. Only one result from the trained DiT model to PixART may not be enough as the answer to question 1. As the authors motivate this work with question 1 & 2, introducing more related analysis would be helpful.
3. It seems like the data construction pipeline could be improved by introducing images in reality as [1]. Given that you use VLMs for data refinement, it may be beneficial to use images in reality as ground truth and use VLMs to generation prompts.
4. In Section 4.3, models trained without Discrimination & Expansion & Mutation (first line) can gain comparable results as models trained with Discrimination & Expansion & Mutation (last line). This may cause confusion, as it may lead to a questions: would it be necessary to perform Discrimination & Expansion & Mutation? Would it be all we need to just query advanced generative models to get data?

[1] Zhao, Haozhe, et al. "UltraEdit: Instruction-based Fine-Grained Image Editing at Scale." arXiv preprint arXiv:2407.05282 (2024).
questions: questions: See weakness.
limitations: limitations: Yes, this paper has limitation section.

Comments: title: Reply to Reviewer i5a7
Reply to Reviewer gzrb
Reply to Reviewer gzrb
Final Rating
comment: Dear Reviewer i5a7,

Thank you for your recognition and for providing insightful comments.

Best regards,

Authors of Paper 1029.
Dear Reviewer gzrb,

To further address your concerns, we provide detailed text prompts for mutation as follows, which will be included in the revised paper.

Text Prompt:

'Now, exercise your imagination to generate 1 new text description for visual contents, {*enhanced_prompt*}. It should be completely unrelated to the previous images and have a completely different structure from the previous text descriptions. Arrange it in the format of a list ["Text description"].'

The *enhanced_prompt* is a prompt controlling the length and the granular level of the generated samples.

The *enhanced_prompt* is randomly sampled from the following ones：
- 'which should contain less than 10 words and be rough and short'
- 'which should contain less than 30 words with different granular levels of details'
- 'which should contain over 30 words with a lot of details'

Three corresponding generated results are as follows：

- ["The sun rises over a calm sea."]

- ["A lone adventurer stands at the edge of a cliff, gazing into the distance, with a single white bird flying overhead and a black raven perched on a nearby rock."]

- ["In the serene setting of a lush garden, a group of vibrant flowers and a variety of exotic fruits coexist harmoniously. The garden is teeming with life, from the delicate petals of the flowers to the succulent fruits hanging from the trees. The colors are a feast for the eyes, with the reds, blues, and yellows of the flowers contrasting beautifully against the green foliage. The fruits, in shades of red, orange, and yellow, add a pop of color to the scene. The garden is a symphony of nature, where every element has its place and purpose."]

Best regards,

Authors of Paper 1029.
Dear Reviewer gzrb,

Thanks for your valuable feedback. Yes, we will add the details in the response and provide more explanations of mutation in the main paper. The detailed prompts we use to instruct the VLM will be provided in the appendix. Thanks again for your comments on improving our paper.

Best regards,

Authors of Paper 1029.
Thank you for your response. I would appreciate adding more details into the paper about the sources of data for all uses, as you explained in response to Q1. In addition, I believe the paper would benefit from more details on the Mutation operation, since at the moment there is only one line describing the operation. For example, what prompt(s) do you use to get the LLM to generate entirely new T2I prompts?

Assuming this information is added to the paper/appendix, I am willing to raise my score to 7.
Thanks for the response. The additional experiments address my concerns. I'll keep my rating as strong accept (8).
Dear reviewers: as you are aware, the reviewer-author discussions phase ends on Aug 13. We request you to kindly make use of the remaining time to contribute productively to these discussions. If you have not read and/or responded to author rebuttal, please do it asap so that the authors get a chance to respond to you. If you have more questions to ask or want further clarification from the authors, please feel free to do it.