SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data

Authors: jialu  li
jaemin  cho
yi-lin  sung
jaehong  yoon
mohit  bansal
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:35:39

Abstract: Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fail to generate images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM’s in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models. We provide code in the supplementary materials.

Main paper: 
 introduction: 
 Text-to-Image (T2I) generation models have shown impressive development in recent years [61; 59; 50; 30; 57; 85; 10] . Although these approaches can generate high-quality images based on textual inputs, they still struggle to capture all semantics in the given textual prompts, such as failing to compose multiple subjects [85; 22; 40] and generate correct spatial relationships [48] .Many recent works have been proposed to tackle these challenges in text-to-image generation, aiming to enhance the faithfulness of T2I models to textual inputs. One line of research focuses on supervised fine-tuning on high-quality image-text datasets with human annotations [18] or image-text pairs with re-captioned text prompts [65; 5] , as shown in Fig. 1 (a) . Another line of research is based on aligning T2I models with human preference annotations [82; 52; 21; 33; 76] , as shown in Fig. 1 "An elegant room with floorto-ceiling bookshelves, filled with an impressive collection of books of all genres. The cozy reading nook by the window invites anyone to curl up with a good book.""A cat with wings flying over a field of donuts." works focus on introducing additional layouts or object grounding boxes to guide the generation process [37; 81; 84; 22; 16; 89] . Despite achieving significant improvements in aligning generated images with input textual prompts, the success of these approaches relies on the quality of the layouts created from the textual prompts, the collection of high-quality annotations with human efforts, or the existence of large-scale ground truth data, which involves expensive human annotation.Motivated by LLMs' impressive text generation capability (given open-ended task instructions and in-context examples), and recent T2I models' capability in generating highly realistic photos (based on text prompts), we investigate an interesting question to further improve the faithfulness of state-of-the-art T2I models: "Can we automatically generate multi-skill image-text datasets with LLMs and T2I models, to effectively and efficiently teach different image generation skills to T2I models?" In this paper, we propose SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm for eliciting the pre-trained knowledge in T2I models for improved faithfulness based on skill-specific learning and merging of experts. SELMA consists of four stages: (1) collecting skill-specific prompts with in-context learning of LLMs, (2) self-generating image-text samples for diverse skills without the need of human annotation nor feedback from reward models, (3) fine-tuning the expert T2I models on these datasets separately, and (4) obtaining the final model by merging experts of each dataset for efficient adaptation to different skills and mitigation of knowledge conflict in joint training. We illustrate the SELMA pipeline in Fig. 1 (c ).In the first and second stages, we use the LLM and the T2I model to generate skill-specific imagetext data. The skills include understanding common objects (e.g., puppy in a backyard), handling long prompts (e.g., an elegant room with floor-to-ceiling bookshelves, filled with an impressive collection of books of all genres. The cozy reading nook by the window invites anyone to curl up with a good book."), and displaying commonsense-defying scenes (e.g., cat flying over sky). We aim to teach diverse generation skills to the same T2I model (i.e., self-learning), so that they can handle different types of prompts. To generate image-text pairs for different skills, we first query GPT-3.5 [46] for prompt generation by using only three skill-specific prompts as in-context examples, and filter the generated prompts with ROUGE-L score to maximize prompt diversity to collect 1K prompts in total (Sec. 3.1). Then we use Stable Diffusion models [59; 50] themselves to generate corresponding images from the prompts (Sec. 3.2) . We find that our skill-specific training can help mitigate knowledge conflict when jointly learning multiple skills (see Table 2 ).In the third and fourth stages, we fine-tune a T2I model with the collected image-text pairs to teach different skills. However, updating the entire model weights can be inefficient; knowledge conflicts within mixed datasets may also lead to suboptimal performance [42] . Thus, in the third stage, we finetune T2I models on these self-generated image-text pairs with parameter-efficient LoRA (low-rank adaptation) modules [27] to create skill-specific expert T2I models (Sec. 3.3). In the fourth stage, to build a joint multi-skill T2I model that can have faithful generations across different skills, we merge the skill-specific experts based on LoRA merging [66; 91] (Sec. 3.4).We validate the usefulness of SELMA with public state-of-the-art T2I models -a family of Stable Diffusion -v1.4 [59] , v2 [59] , and XL [50] on two text faithfulness evaluation benchmarks (DSG [14] and TIFA [28] ), three human preference metrics (Pick-a-Pic [31] , ImageReward [82] , and HPS [79] ), and human evaluation. Empirical results demonstrate that SELMA significantly improves T2I models' faithfulness to input text prompts and achieves higher human preference metrics. Our final LoRA-Merging model achieves 6.9% improvements on DSG, 2.1% improvements on TIFA, and improves the human preference metrics by 0.4 on Pick-a-Pic, 0.39 on ImageReward, and 3.7 on HPS. Furthermore, we empirically show that the T2I models learned from the self-generated images achieve a performance similar to that of learning from ground-truth images (see Fig. 3 ). Lastly, we further show that fine-tuning with images from a weaker T2I model (i.e., SD v2) can help improve the faithfulness of a stronger T2I model (i.e., SDXL), suggesting promising weak-to-strong generalization in text-to-image models (see Table 3 ). 
related work: 
 Training Vision-Language Models with Synthetic Images. As recent denoising diffusion models [69; 25] have achieved photorealistic image synthesis capabilities, many works have studied using their synthetic images for training different models. Azizi et al. [4] , Sariyildiz et al. [63] , Lei et al. [34] , inter alia, study training image classification models with synthetic images. For image captioning, Caffagni et al. [9] use diffusion models to generate images on the captioning data. For training CLIP [53] models, several works use diffusion models to generate images from existing captions [74] or text generated with language models [23] . There is a recent research direction using synthetic images to train image generation models themselves, and we discuss more details in the following paragraph.Training Text-to-Image Generation Models with Synthetic Images. A line of recent works train text-to-image (T2I) generation models with synthetic images generated by the same or other models annotated with human preference scores using reinforcement learning [33; 82; 79; 20; 17; 21] or direct preference optimization (DPO) [55; 76] . While these works show promising results in improving model behavior with human preferences, they require expensive human preference annotations. SPIN-Diffusion [87] proposes using self-play [62; 73] , which was successfully adopted in Alphago Zero [67] and language models [13; 88] , where the model itself becomes a judge and iteratively compares itself with previous iterations. However, self-play algorithm still relies on a set of ground truth image-text pairs as positive examples for supervision. Concurrent/independent to our work, DreamSync [70] trains a T2I model by first creating text prompts with LLMs, sampling multiple images by the T2I model itself, filtering out images with off-the-shelf scorers, and fine-tuning the model on the resulting synthetic image-text pairs [70] . Unlike DreamSync that depends on image filtering (generating 8 images and taking at most one of them for each text prompt, SELMA generates 1 image for each prompt), significantly improving data generation efficiency by using only 2% of image-text pairs compared with DreamSync. Furthermore, we focus on learning multiple skills with T2I models by learning and merging skill-specific LoRA experts to mitigate knowledge interference across different skills, and we show this approach attains much stronger performance without adding any additional inference cost (see Table 2 ). 
selma: learning and merging text-to-image skill-specific experts with: 
 Auto-Generated DataWe introduce SELMA, a novel framework to teach different skills to a T2I generation model based on auto-generated data and model merging. As illustrated in Fig. 2 , SELMA consists of four stages:(1) skill-specific prompt generation with LLM (Sec. As shown in Fig. 2 (a), we automatically collect skill-specific prompts (that will be paired with images in Sec. 3.2) to fine-tune T2I models in two steps: (1) using large language models (LLMs) to generate prompts with brief skill descriptions and a few example prompts and (2) filtering the generated prompts to ensure their diversity. In the following, we explain the two steps in detail.Prompt Generation. We leverage the in-context learning ability of LLMs to generate additional text prompts that follow similar writing styles (e.g., paragraph style) or acquire models' knowledge in the same domain (e.g., count capability). We manually collect three seed prompts with similar writing styles or acquire similar skills (e.g., spatial reasoning) to the target text prompts. Next, we use these seed prompts as in-context learning examples to query GPT-3.5 (GPT3.5-turbo-instruct) [46] . We provide additional instructions that encourage diversity in generated prompts, including object occurrences, sentence patterns, and required skills for the T2I model to generate accurate prompts. The detailed prompt template can be found in the Appendix. During prompt generation, we keep expanding the seed prompts with the generated prompts, and always randomly sample three prompts as in-context learning examples from the seed prompts.Prompt Filtering. To improve the diversity of the collected text prompts, we filter out prompts that are similar to already generated ones. As Taori et al. [72] demonstrate that instruction diversity is crucial for improving the instruction following capability of large language models, we follow the same intuition to create diverse text prompts. To ensure the diversity of generated prompts, we first receive a newly generated text prompt from the previous step. Then, we calculate its highest ROUGE-L [38] score with all the previously generated and filtered prompts. Following Taori et al. [72] , we discard text prompts with ROUGE-L>0.8 to maximize the diversity of generated prompts.As illustrated in Fig. 2 (b), we generate corresponding images for each generated text prompt using the T2I model. We find that existing diffusion-based T2I models are highly effective in learning from their self-generated images, and even benefit from learning from images generated with weaker T2I models (Table 3 ). It is important to leverage the knowledge that already exists inside the T2I models (learned from web data during pre-training), and hence we aim to extract this knowledge for creating the skill-specific image-text pairs, and use them to improve T2I models' faithfulness (Sec. 3.3).We efficiently adapt the T2I model to different skills by learning skill-specific Low-Rank Adaptation (LoRA) [27] experts. In LoRA fine-tuning, the updates to the original weights W 0 ∈ R d×d is decomposed with two low-rank matrices: W 0 + ∆W = W 0 + BA, where B ∈ R d×r , A ∈ R r×d and r << d. For each new dataset, we fine-tune the T2I model with LoRA independently, and this introduces N skill-specific LoRA experts (as shown in Fig. 2 (c )). In Sec. 5.2, we observe that learning and merging skill-specific experts is more effective than learning a single LoRA across all datasets, by helping the T2I model mitigate knowledge conflicts between different skills [42; 11] . However, using multiple skill-specific experts requires the model to know which expert to use for a given input, and this usually requires user annotations on the skill category of inputs. In the next section, we propose to merge skill-specific experts to efficiently construct a single multi-skill model. ; the resulting single expert can handle all N skills simultaneously (as shown in Fig. 2 (d) ). With this approach, we can reach superior performance over standard multi-task LoRA training and even MoE-LoRA (learning a router with LoRA experts), as shown in Tables 2 and 5 , and also eliminate the need to know the skill categories beforehand. Note that while ZipLoRA [66] has demonstrated the use of LoRA merging (merging 2 LoRA modules) in diffusion models, to the best of our knowledge, we are the first to show the effectiveness of LoRA merging on multiple diverse skills (from 5 datasets) in diffusion models.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We include code in the supplemental material. Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setup 4.1 evaluation benchmarks: 
 We evaluate models on two evaluation benchmarks that measure the alignment between text prompts and generated images: DSG [14] and TIFA [28] .DSG consists of 1060 prompts from 10 different sources (160 prompts from TIFA [28] , and 100 prompts from each of Localized Narratives [51] , DiffusionDB [77] , CountBench [47] , Whoops [6] , DrawText [41] , Midjourney [75] , Stanford Paragraph [32] , VRD [43] , PoseScript [19] ). Among the ten DSG prompt sources, we mainly experiment with text prompts from five prompt sources that have (1) ground-truth image-text pairs (to compare the usefulness of auto-generated data with ground-truth data) and (2) measuring different skills required in T2I generation (e.g., following long captions, composing infrequent objects). Specifically, we use COCO [39] for short prompts with common objects in daily life, Localized Narratives [51] for paragraph-style long captions, DiffusionDB [77] for human-written prompts that specify many attribute details, CountBench [47] for evaluating object counting, and Whoops [6] for commonsense-defying text prompts.TIFA consists of 4,081 prompts from four sources, including COCO [39] for short prompts with common objects, PartiPrompts [85] / DrawBench [61] for challenging image generation skills, and PaintSkills [15] for compositional visual reasoning skills.We quantitatively evaluate the performance of T2I generation models in text faithfulness and human preference metrics. Specifically, to evaluate text faithfulness, we use VQA accuracy from TIFA [28] and DSG [14] . To evaluate human preference score, we use the PickScore [31] , ImageReward [82] , and HPS [79] See also Sec. 5.6 for human evaluation. Details can be found in Appendix.In the prompt generation stage (Sec. 3.1), we use gpt-3.5-turbo-instruct [46] to generate text prompts. We collect 1K prompts for each of the five datasets (COCO [39] , Localized Narratives [51] , DiffusionDB [77] , CountBench [47] , and Whoops [6] ). We refer to the resulting auto-generated datasets as Localized Narrative SELMA , CountBench SELMA , DiffusionDB SELMA , Whoops SELMA , and COCO SELMA , and the resulting combination of 5K auto-generated dataset as DSG SELMA-5K .In the image generation stage (Sec. 3.2), we use the default denoising steps 50 for all models, and the Classifier-Free Guidance (CFG) [26] of 7.5. In the LoRA fine-tuning stage (Sec. 3.3), we use 128 as the LoRA rank. During inference, we uniformly merge the specialized LoRA experts into one multi-skill expert (Sec. 3.4). More details can be found in Appendix. 5 Results and Analysis 
comparison with different alignment methods for text-to-image generation: 
 We compare SELMA with different alignment methods for T2I generation, including training-free methods (SynGen [58] , StructureDiffusion [22] ), RL-based methods (DPOK [21] , DDPO [7] ), and DreamSync [70] , a concurrent method based on automatic data generation. We experiment with three diffusion-based T2I models (i.e., SD v1.4, SD v2, and SDXL).SELMA outperforms other alignment methods for T2I generation. As shown in Table 1 , SELMA consistently improves faithfulness and human preference metrics for all three backbones. Specifically, on SD v1.4, SELMA improves the baseline by 2.9% in TIFA, 4.0% in DSG, 0.2 in PickScore, 0.58 in ImageReward, and 2.5 in HPS score. Furthermore, SELMA achieves significantly higher performance than other baselines, including the RL-based methods (DPOK/DDPO), which require annotated human preference data, and DreamSync, a concurrent/independent work based on a larger auto-generated dataset (i.e., 28K text prompts; SELMA uses 5K text training prompts in total), and image filtering (i.e., generating 8 images and taking at most one of them for each text prompt; SELMA only generates 1 image for each prompt). Besides, on SD v2 and SDXL, SELMA shows larger improvement in text faithfulness (i.e., 7.4% improvement on DSG for SD v2, and 6.9% on DSG for SDXL), demonstrating the effectiveness of SELMA.We compare (1) separately learning multiple LoRA experts on different auto-generated datasets followed by merging and ( 2) training a single LoRA on a mixture of datasets. For this, we experiment with our five auto-generated image-text pairs: Localized Narrative SELMA , CountBench SELMA , DiffusionDB SELMA , Whoops SELMA , and COCO SELMA (see Sec. 4.3 for details).Learning & merging skill-specific LoRA experts is more effective than single LoRA on multiple datasets. Table 2 shows that the LoRA models trained separately on each of the five automatically generated datasets (No.1. to No.5.) can improve the overall metric over the baseline SD v2 -70.3%, while the degree of improvements is different for each metric (e.g., 76.4% for fine-tuning with Localized Narrative SELMA , and 73.0% for fine-tuning with DiffusionDB SELMA ). However, training multiple skills simultaneously with a single LoRA (No.6. to No.7.) tends to degrade performance as more datasets are incorporated. This indicates that the T2I model struggles with LoRA to accommodate distinct skills and writing styles from different datasets. A similar phenomenon has been reported in LLaVA-MoLE [11] , where the knowledge conflict between multiple datasets can degrade the performance. We're the first to show this knowledge conflict across different skills also exists in diffusion models. We find that merging multiple skill-specific LoRA experts (No.8. and No.9.) achieves the best performance in both text faithfulness and human preference, demonstrating that merging LoRA experts can help mitigate the knowledge conflict between multiple skills.In this section, we investigate the effectiveness of our automatically generated data by comparing them with ground truth data. We fine-tune SD v2 model using ground truth data from Localized Narratives, CountBench, DiffusionDB, Whoops, and COCO, sampling 1K image-text pairs from each dataset and fine-tuning specialized LoRA experts accordingly. Fine-tuning with auto-generated data can achieve comparable performance to fine-tuning with ground truth data. As shown in Fig. 3 , we observe that fine-tuning with either auto-generated or ground truth data improves from baseline SD v2 performance -70.3%, when evaluated on the DSG benchmark. Surprisingly, fine-tuning with the generated data via SELMA outperforms the use of ground truth data in most cases, leading to a DSG accuracy improvement of 4.0% with Localized Narrative style prompts, 1.0% with Count-Bench style prompts, 1.9% with Dif-fusionDB style prompts, and 0.9% with COCO style prompts. In short, our approach results in an average improvement of 1.2% brought by fine-tuning only auto-generated data without any need for human-collected ground truth text-image pairs, suggesting that diffusion-based text-to-image models may benefit from the diversity of self-generated images. Furthermore, we investigate whether the improvement is brought by text prompt or image quality. We generate images with SD v2 based on 1K ground truth captions, and fine-tune specialized LoRA experts accordingly. We observe that in most cases, using generated images works better than ground truth images (e.g., Localized Narrative), suggesting T2I models can generate images with comparable alignment as ground truth images. Besides, learning from our LLM-generated captions achieves comparable performance with learning from ground truth captions, suggesting the effectiveness of our text prompt collection process. Lastly, we also show in Appendix that fine-tuning with LLaMA3 [1] generated prompts also improve T2I models' faithfulness in generation, demonstrating that our proposed SELMA is compatible to different LLM-based prompt generator. Weaker T2I models can help stronger T2I models. As shown in Table 3 , fine-tuning SDXL with generated images from SD v2 (No.4.) remarkably enhances performance over the SDXL baseline (No.2.) in both text faithfulness and human preference. In addition, this approach achieves competitive performance compared with fine-tuning SDXL with SDXL-generated images (No.5.), indicating a promising potential for weak-to-strong generalization in diffusion-based T2I generation models. To the best of our knowledge, this is the first work to find promising improvements in the weak-to-strong generalization for text-to-image diffusion models. In this section, we demonstrate that our proposed paradigm is compatible with different prompt generator LLMs. Specifically, we experiment with LLaMA3 (8B) [1] , a publicly available open-source LLM and compare the results with GPT-3.5 (gpt-3.5-turbo-instruct) based setups described in the main paper.With both LLMs, we generate five sets of skill-specific prompts, and each set contains one thousand skill-specific prompts. As shown in Table 4 , we find that fine-tuning SDXL with data generated with LLaMA3 achieves 78.6% on average on DSG, improving the baseline by 5.3%, closing the gap to GPT-3.5 based results. This demonstrates that SELMA is flexible and compatible with different prompt generator LLMs. Besides, we further experiment with fine-tuning SDXL with data generated with both a weaker image generator SDv2 and a weaker prompt generator LLaMA3. Our results show that this model achieves similar performance as the model fine-tuned with images generated with SDXL, demonstrating that weak-to-strong generalization holds with weaker data generators.In addition to automatic evaluation using text faithfulness benchmarks (DSG and TIFA) and human preference metrics (PickScore, ImageReward, and HPS), we further perform a human evaluation to compare the performance of SDXL and SDXL fine-tuned with SELMA on DSG SELMA-5K (details in Sec. 4.3). We randomly select 200 prompts from DSG and ask three annotators to determine "Which image aligns with the caption better?" given the text prompt and generated images from both SDXL and SDXL+SELMA. We provide win/tie/lose options to the annotators, and we report the win vs. lose percentage in the following. The user interface, instructions, and the detailed statistics are provided in appendix. SDXL+SELMA is preferred than SDXL in terms of text alignment. Fig. 4 shows that on all five DSG splits, images generated with SDXL+SELMA are preferred 67.9% of the time, compared to 32.1% for the baseline SDXL. Furthermore, on the five datasets fine-tuned with similar text prompts, SDXL+SELMA achieve a preference rate of 94.6% on COCO split and 79.2% on Localized Narratives. This substantial preference over the baseline model demonstrates the effectiveness of SELMA in enhancing T2I models. We experiment with various training configurations for SELMA to validate our design choices for fine-tuning. As our current experiments are based on supervised fine-tuning with LoRA Merging, we additionally explore Direct Preference Optimization (DPO) [55; 76] as an alternative to supervised fine-tuning and employing Mixture of Lora Experts (MoE-LoRA) [80] instead of LoRA Merging. See the Appendix for the implementation details. Table 5 demonstrates that while fine-tuning with DPO and MoE-LoRA significantly improves the T2I models' text faithfulness and human preference (No.2 & 3. vs. No.0.), simple inference-time LoRA merging achieves the best overall performance. In the end, we adopt LoRA merging and supervised fine-tuning as the default configuration in SELMA for its simplicity and efficiency. In this section, we demonstrate that simply scaling the LoRA size doesn't help mitigate the knowledge conflict between different skills. We experiment with using a single LoRA with different ranks (128 / 256 / 640) and compare them to our default LoRA merging (rank=128). Table 6 shows that increasing the rank of LoRA from 128 to 256 slightly improves the performance (i.e., 74.9 vs. 74.4), but further scaling the rank of the LoRA to 640 significantly drops the performance (i.e., 71.5 vs. 74.4). The performance drop when using LoRA with higher ranks (i.e., rank=640) is similar to the observation in Figure 3 in [24] . This result indicates the effectiveness of our skill-specific learning and merging of LoRA experts.Figure 5 : Example images generated with SDXL and SDXL+SELMA. SELMA shows better performance in object composition, attribute binding, and long text prompt following. We highlight the parts of the prompts in red where SDXL makes errors while SDXL+SELMA generates correctly.We show some qualitative examples of images generated with SDXL fine-tuned with SELMA paradigm in Fig. 5 . We find that fine-tuning with SELMA improves SDXL's capability in composing infrequently co-occurred attributes (i.e., "cube" and "denim" in the top-left image), composing multiple objects mentioned in the text prompts (i.e., "brown and white cake", "table", "red and yellow flower", and "fork" in the top-right image), following details in the long paragraph-style text prompts (i.e., "blue train with white stripes" and "long yellow line near train area" in the bottom-left image), and generating images that challenge commonsense (i.e., "Old man lifts a barbell" in bottom-right image). These qualitative examples demonstrate the effectiveness of SELMA in improving T2I models' text faithfulness and human preference. 
conclusion: 
 We propose SELMA, an novel paradigm to improve state-of-the-art T2I models' faithfulness in generation and human preference by eliciting the pre-trained knowledge of T2I models. SELMA first collects self-generated images given diverse generated text prompts without the need for additional human annotation. Then, SELMA fine-tunes separate LoRA models on different datasets and merges them during inference to mitigate knowledge conflict between datasets. SELMA demonstrates strong empirical results in improving T2I models' faithfulness and alignments to human preference and suggests potential weak-to-strong generalization for diffusion-based T2I models. Figure 6 : Qualitative example images generated with SDXL and SDXL+SELMA (Ours) from DSG [14] test prompts requiring different skills. SELMA helps improve SDXL in various skills, including counting, text rendering, spatial relationships, and attribute binding. We highlight the parts of the prompts in red where SDXL makes errors while SDXL+SELMA generates correctly. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We include implementation details in main paper Sec. 4.3, and further add more details in Appendix H.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We report single-run results for all the experiments following previous work.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: This research conforms every respect with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts in Appendix K.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We don't have such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited previous work properly in Appendix L and strictly followed the license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In Fig. 6 , we show additional qualitative examples of SDXL and SDXL+SELMA from DSG [14] test prompts requiring different skills. SELMA helps improve SDXL in various skills, including counting, text rendering, spatial relationships, and attribute binding. For counting skill prompts, SDXL+SELMA generates "four bees" and "seven groomsmen" correctly following the text prompts.For text rendering skill prompts, SDXL+SELMA can render the text ("knowledge is power" and "Fall is here") more accurately, while it still lacks the capability to render the text in the texture of vines or autumn leaves. For entity skill (placing correct objects) prompts, the SDXL sometimes misses some entities mentioned in the text prompt (i.e., "A child", and "A road sign"), while SDXL+SELMA can successfully generate them. For spatial relationship skill prompts, SDXL+SELMA generated images (i.e., "holding a stick in its mouth", and "stop sign left to dog"). Lastly, for attribute skill prompts, SDXL+SELMA binds objects with their corresponding attributes (i.e., "cotton candy trees" and "A man in black suit") more accurately than SDXL. These qualitative results demonstrate the effectiveness of SELMA.In this section, we show the detailed VQA accuracy for each skill category on TIFA and DSG benchmarks. Since DreamSync [70] does not provide the skill-specific scores, we report the skillspecific scores of SD models on TIFA and DSG and based on our experiments in Table 8 and Table 9 ; we observe there are less than 1% score differences of SD/SDXL models in TIFA average accuracy between the results in Sun et al. and ours.As shown in Table 8 and Table 9 , on both TIFA and DSG benchmarks, SELMA improves the generation faithfulness in most of the categories. Comparing SDXL and SDXL+SELMA, the SDXL finetuned with SELMA approach shows large improvement especially in entity (i.e., 3.1% on animal/human on TIFA compared with SDXL, 5.6% in whole on DSG, 12.2% in part on DSG), as well as spatial relationship (i.e., 1.9% on TIFA, and 8.0% on DSG), and counting skills (i.e., 1.4% on TIFA, and 13.2% on DSG). Besides, we also observe that SELMA significantly improves the text rendering for SD v2 and SDXL (i.e., 16.4% compared with SDXL, and 6.5% compared with SD v2 on DSG), but not SD v1.4 (i.e., 1.8% decrease compared with SD v1.4 on DSG). We conduct the human evaluation (described in the main paper Sec. 5.5) on 200 randomly sampled prompts from DSG, with three external annotators. We show the annotation interface in Fig. 7 . The image order between the two models is randomized to avoid the leakage of information about which image is generated with which model.In Table 10 , we show the detailed annotator votes for win, lose, and tie for SDXL and SDXL+SELMA. SDXL+SELMA has significantly higher win votes compared with SDXL on all the 200 sampled text prompts (i.e., 241 win vs. 114 lose), demonstrating the effectiveness of SELMA.We demonstrate that our automatically collected data is of high quality to improve the faithfulness of T2I models. Specifically, we filter out the generated images based on TIFA score (>0.9) and VILA score (>0.6), and maintain 1K data for each set of skill-specific prompts. Fine-tuning with filtered images achieves 80.3% DSG score with SDXL backbone, which is similar performance as fine-tuning with un-filtered version (i.e., 80.2% on DSG), indicating the overall high quality of the data, which does not lead to performance degradation.In this section, we investigate whether fine-tuning with T2I-generated images will increase the bias in existing T2I models. Specifically, we evaluate the T2I model with GEP score [90] , which measures the gender bias in T2I models. Specifically, a learned cross-modal classifier is used to distinguish attributes (e.g., dress) in the image, and GEP score calculates attribute-wise difference based on output score from the cross-modal classifier. A lower GEP score indicates less gender bias. Fine-tuning SD v2 with our generated data achieves 0.04413 GEP score, while fine-tuning with ground truth data achieves 0.04976. This indicates that fine-tuning with our generated data does not amplify the bias in existing T2I models compared with fine-tuning with ground truth data.In the prompt generation stage (Sec. 3.1), we use gpt-3.5-turbo-instruct [46] to generate text prompts by providing three prompts for each skill as in-context examples. For each of the five datasets (COCO [39] , Localized Narratives [51] , DiffusionDB [77] , CountBench [47] , and Whoops [6] ), we collect 1K prompts starting with three prompts randomly sampled from them, ensuring the prompts are not included in the DSG test prompts (i.e., 5K prompts in total). We refer to the resulting auto-generated datasets as Localized Narrative SELMA , CountBench SELMA , DiffusionDB SELMA , Whoops SELMA , and COCO SELMA . We refer to the resulting combination of 5K auto-generated dataset as DSG SELMA-5K .In the image generation stage (Sec. 3.2), we use the default denoising steps 50 for all models, and the Classifier-Free Guidance (CFG) [26] of 7.5. In the LoRA fine-tuning stage (Sec. 3.3), we use 128 as the LoRA rank. We fine-tune LoRA in mixed precision (i.e., FP16) with a constant learning rate of 3e-4 and a batch size of 64. We fine-tune LoRA modules for 5000 steps, which is approximately 313 epochs. During inference, we uniformly merge the specialized LoRA experts into one multi-skill expert (Sec. 3.4). We evaluate model checkpoints every 1000 steps and pick the model with the best text faithfulness on DSG benchmark. Fine-tuning LoRA for SD v1.4, SD v2, and SDXL takes 6 hours, 6 hours, and 12 hours on a single NVIDIA L40 GPU, respectively. We use Diffusers [49] for our experiments.In this section, we provide the implementation details of two training approaches we experiment with (described in Sec. 5.6 in the main paper).Direct Preference Optimization (DPO). We fine-tune LoRA models with DPO proposed in [76] . Specifically, we sample two images with T2I models and calculate the image-text alignment with CLIP score [54] . We use the image with a higher CLIP score as the positive example and the image with a lower CLIP score as the negative example. We fine-tune the LoRA models to learn to generate images closer to the positive image distribution and push away from generating images similar to the negative image distribution. Similarly, we fine-tune with DPO on five datasets with different text styles and skills and merge LoRA expert models during inference time by averaging the LoRA weights. In DPO training, we use a constant learning rate 3e-4 and fine-tune LoRA for 5K steps. We evaluate the model on DSG every 1K steps and pick the best checkpoint.Mixture of Lora Experts (MoE-LoRA). MoE-LoRA [80] utilizes a gating function (router) to decide which experts to use during training and inference. The gating function predicts weights for each expert based on layer inputs and picks the top K experts to use at each layer. Specifically, the gating function we use is a simple linear mapping function, where {w i } K i=1 = W g x. x is the input to each layer, W g is the learnable gating weights, and {w i } K i=1 are the predicted weights. The outputs of each expert are added together with the normalized weights from the gating function. In MoE-LoRA, we initialize five LoRA experts fine-tuned on different datasets containing different text styles and skills. We freeze the learned LoRA weights and only fine-tune the gating function on the collected five datasets. We also compare with learning LoRA weights along with the router, which achieves worse performance (75.9 on DSG). We activate all five experts during training and inference (i.e., K = 5). In MoE-LoRA training, we use a constant learning rate 1e-5 and fine-tune LoRA for 5K steps. We evaluate the model on DSG every 1K steps and pick the model with highest text faithfulness score. We show the prompts we use to query GPT3.5 to generate 1K prompts for each skill. As shown in Fig. 8 , we use different prompts to generate SELMA data. For example, we specify "paragraph captions" to generate text prompts that can be used to teach model to follow long text prompts, and specifying "evaluate model's capability to count objects" to collect a set of prompts for improving model's counting capability. Besides, in all the prompt generation, we emphasize that the image should "mention diverse objects" to maximize the semantic diversity in generated prompts.Text-to-image generation models can be used in many real-world applications, such as creating content for media and entertainment. Our proposed SELMA improves T2I models' faithfulness with auto-generated data, reducing human annotation efforts for high-quality image-text pairs. However, we also note that our SELMA has several limitations. SELMA relies on a strong image generator and an instruction-following LLM. Note that SELMA is model-agnostic and can be implemented with publicly accessible models (GPT-3.5/LLaMA3 and Stable Diffusion models). Also, since our fine-tuning works well with a small number of image-text pairs (i.e., for each skill, we only generate 1K text prompts and generating one image per each text prompt), the cost of LLM inference (i.e., $27.78 for querying GPT-3.5 for generating prompts in all the experiments including ablation studies) and image generation (8s per image for image generation with SDXL on a single NVIDIA L40 GPU with 48GB memory) is minimal. Besides, although SELMA helps boost T2I models' performance significantly in both text faithfulness and alignment to human preference, fine-tuning with SELMA does not guarantee the resulting model to follow the text prompts in every detail. To use T2I models trained with SELMA, researchers should first carefully study their capabilities in relation to the specific context in which they are being applied.We provide the licenses of the existing assets we use in this paper in Table 11 .Table 11 : A list of the licenses of the existing assets used in this paper.PyTorch [3] BSD-style Huggingface Transformers [78] Apache License 2.0 Torchvision [44] BSD 3-Clause "New" or "Revised" License Diffusers [49] Apache License 2.0 Stable Diffusion [59] CreativeML Open RAIL-M COCO dataset [39] CC BY 4.0 Localized Narrative dataset [51] CC BY 4.0 DiffusionDB [77] MIT License Whoops [6] CC BY 4.0 CountBench [47] (LAION-400M [64] subset) CC BY 4.0 GPT3.5 [46] OpenAI Terms of Use LLaMA3 [1] Meta LLaMA3 LicenseNeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Benchmark performance can be found: https://github.com/tgxs002/HPSv2 


Decision: AC, along with the reviewers, holds a positive opinion towards SELMA. Although the novelty level is being questioned at first, the authors made good attempt to argue this way of using LLM's capability to conduct skill specific text 2 image data augmentation is unique and novel. While presentation could be improved, as suggested by multiple reviewers, AC thinks the current way of presentation, as it is simply a very engineering idea, is already decent. AC strongly suggests the authors take the review comments, especially WQ94's weakness point, into the final draft's revision. Overall, I would recommend Accept (poster).

Reviews: review: paper_summary: paper_summary: The goal of this paper is to improve the faithfulness of text-to-image generation models. New datasets and fine-tuning frameworks are introduced to address this limitation. Specifically, this paper first adopts LLMs to generate multiple datasets of text prompts that can teach different skills and then generates images with a T2I image based on the text prompts. In the second stage, the LoRA finetuning is performed on the generated datasets to get different experts. Finally, different LoRA experts are merged into one better T2I model. The experiments show better performance on public benchmark datasets.
strength_weakness: strength_weakness: 1. The technical novelty is a bit weak. LoRA fine-tuning and merging experts are not original. The prompt generation seems to be straightforward too. 

2. Clarity of this paper should be improved. What is the definition of "Skill"? Is it referred to different prompt styles? Why do the self-generated images improve the T2I model, especially when the T2I model fails to generate faithful images?
questions: questions: See the weakness.
limitations: limitations: Limitations are discussed in the appendix!
paper_summary: paper_summary: The paper analyzes the merging of skill-specific LoRA-finetuned models, trained on generated data and compares this approach to other training approaches (i.e. finetuning, PPO). Moreover, the paper compares the use of GT images and prompts to the use of generated data. The results suggest that this approach performs better than the baseline approaches. Finally, the paper shows a weak-to-strong generalization from weaker models.
strength_weakness: strength_weakness: _Weak-to-Strong Generalization_

The claim in this section doesn't mention the fact that the text here comes from an additional strong model. While the generative model is weaker, the use of an LLM here makes this experiment less convincing. One possible way to improve this section is by ablating here separately again the text and the images.  

_Ablation of LoRA parameters_ 

Will a larger $r$ value for the LoRA remove the need for the merging? The paper can benefit from ablation of the size of the bottleneck in the LoRA, together with an experiment that shows that the merging is better than one larger LoRA, trained on all the generated data together. 

_Optimization time comparisons_

An additional concern for the merged LoRAs approach is its optimization time. Can you provide the comparisons for the training time and iteration number for the presented approach and baselines?
questions: questions: See weaknesses.
limitations: limitations: Limitations are provided.
paper_summary: paper_summary: This article goes through four stages: (1) collecting skill-specific prompts using in-context learning of LLMs, (2) self-generating image-text samples for diverse skills without the need for human annotation or feedback from reward models, (3) fine-tuning the expert T2I models on these datasets separately, and (4) obtaining the final model by merging experts from each dataset for efficient adaptation to different skills and mitigation of knowledge conflict in joint training. It is found that the model can optimize itself to an excellent level relying solely on the prompts from the LLM and the model's own generative capabilities.
strength_weakness: strength_weakness: 1. The article does not provide a detailed explanation of the comparison in Table 2 regarding single LoRA. In the comparison between multi-LoRA and single LoRA, are the parameter counts of multi-LoRA and single LoRA the same, or is each LoRA within multi-LoRA equivalent in parameter count to single LoRA? Additionally, is the total training step count for multiple LoRAs in multi-LoRA equal to the training step count for single LoRA, or is the training step count for single LoRA consistent with that of each LoRA within multi-LoRA?

2. The fusion mechanism among multiple LoRAs does not seem to have been thoroughly explored through sufficient ablation experiments. If, after training each LoRA separately in multi-LoRA, a router is introduced to perform gating operations on multi-LoRA, similar to MoE-LoRA, would the effectiveness improve?
questions: questions: See weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes a workflow in which synthetic Text To Image data is generated in order to improve faithfulness of T2I models. Specifically, they generate prompts with LLMs, then Images with T2I models and finally fine-tune pre-trained T2I models with LORA fine-tuning. They fine-tune multiple LORA experts and then merge them by merging parameters. On a limited set of evals, the proposed model achieves good results.
strength_weakness: strength_weakness: The strength mentioned above is also a weakness of the paper. The paper is motivated by faithfulness. However, the proposed method tackles faithfulness at best as a side effect. While this means that the method might be more general, it also negatively impacts the clarity of the paper and ease of understanding. It might be better to just focus on the multi-task fine-tuning task instead of the impact of individual tasks or side-effects.
questions: questions: Given the complex setup of the proposed workflow, there is a large number of design decisions to make. Do you have guidance, experiments which part of the workflow is the most sensitive with respect to downstream results, e.g., LLM choice, in-context example selection, LLM prompt choice, T2I choice, LORA parameters, etc.?
limitations: limitations: The limitations are only discussed in the appendix. I think it would be important to have a comprehensive discussion of limitations in the main text.

Comments: title: Response to authors
Final review
comment: The authors solve my issues and I maintain the rating to accept it.
The rebuttal addressed my concerns as well as other reviewers' concerns. Therefore, I would like to increase the rating to be a Weak Accept!
I would like to thank the authors for their response to both my questions and the questions by the fellow reviewers. After going over the other reviews and considering all the answers, I still believe that the contributions and novelty of the paper are sufficient to slightly pass the bar for acceptance.
Thank you for the clarifications. I decided to keep my score.