SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
Authors: Jialu Li
Jaemin Cho
Yi-Lin Sung
Jaehong Yoon
Mohit Bansal
Venue: NeurIPS 2024 poster (2024)
Keywords: Text-to-Image Generation; LoRA Learning and Merging; Skill-based Experts; Auto-Generated Data
Primary area: machine_vision

Abstract: Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fail to generate images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM’s in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models. We provide code in the supplementary materials.

SELMA: Learning and Merging Skill-Specific Text-to-lmage Experts with Auto-Generated Data
Jialu Li
Jaemin Cho"
Yi-Lin Sung UNC Chapel Hill
Jachong Yoon
Mohit Bansal
(jialuli, jmincho, ylsung, jhyoon, mbansal]@cs unc.
https://selma t2i http:://eelmm-t2..gttbbb.bbb github.
Abstract
Recent text-to-image (T2I) generation models have demonstrated impressive capa- bilities in creating images from text descriptions. However, these T2I generation models often fail to generate images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto- Generated Data, a novel paradigm to improve the faithfulness T21 models by fine-tuning models on automatically generated. multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM's in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multi- ple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build joint multi-skill T2I model that generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-ar the-art T21 dif- fusion models multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Morcover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models.
with images from weaker T21 model can help improve the
Introduction
Text-to-Image (T2I) generation models have shown impressive development in recent years [61; 59; 50; 30; 57; 85; 10]. Although these approaches can generate high-quality images based on textual inputs, they still struggle capture all semantics in the given textual prompts, such as failing to compose multiple subjects [85; 22; 40] and generate correct spatial relationships [48].
Many recent works have been proposed to tackle these challenges in text-to-image generation, aiming enhance the faithfulness of T2I models to textual inputs. One line of research focuses on supervised fine-tuning on high-quality image-text datasets with human annotations re-captioned text prompts [65; 5]. as shown in Fig. (a). Another line of research based on aligning T21 models with human preference annotations [82; 52; 21; 33; 76), as shown in Fig. (b). Other
[18] image-text pairs with
"qual contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

for day
1. Nluto-Generate Image-Text Pairs Teaching Different Skills
ride."
Image-Text
Skill 1
Skill
Skill
common objocts
"A playfel papp choses its cay a backyard."
Text-to-lmage Model
Large Language Model
long test prompts
Text-to-lmaae Model
"An siegant roce with floor. to-cuilor kookahabees, filled with Inpresctus col lertios of tooks of 11l geanee, The mtyy reating nos by the good book."
(a) Supervised Fine-tuning (SFT)
*An attroble zeitrven
Auto Gett--Gnerrted Image Text Pairs
commoneene-defyyee /Saldd denuts."
Human Preference
Text Image Model
2. Learn Skill-Specific Experts
3. Merge Specifi Experts
Scoring/ / Ranking
Skill
FT
T21 Model
TalModal
T21 Model
Model
Merge
Generate
Skill
FT
T21 Model
Feedback
(b) Fine-tuning with Human Preference
(c) SELMA; Spill-Specific Expert Learning & Merging with Auto-Generated Data
Figure 1: Comparison of different fine-tuning paradigms for text-to-image (T2I) generation models.
(a) Supervised Fine-tuning (SFT): T2I model trained with image-text pairs from existing datasets. (b) Fine-tuning with Human Preference (e.g., RL/DPO): humans annotate their preferences on images by ranking/scoring in terms of text alignments, and T21 model is trained to maximize the human preference scores. (c) SELMA: instead of collecting image-text pairs or human preference annotations, we automatically collect image-text pairs for desired skills with LLM and T2I model, and create a multi-skill T2I model by learning and merging skill-specific expert models.
works focus on introducing additional layouts object grounding boxes to guide the generation process [37; 81; 84; 22; 16; 89]. Despite achieving significant improvements in aligning generated images with input textual prompts, the success of these approaches relies on the quality of the layouts created from the textual prompts, the collection of high high-quality annotations with human efforts, or the existence of large-scale ground truth data, which involves expensive human annotation.
Motivated by LLMs' impressive text generation capability (given open-ended task instructions and in-context examples), and recent T2I models' capability in generating highly realistic photos (based on text prompts), we investigate an interesting question to further improve the faithfulness of sfate-of-the-art T2I models: "Can we automatically generale multi-skill image-lext datasets with LLMs and 721 models, to effectively and efficiently teach different image generation skills to T21 models?" In this paper, we propose SELMA: Skill-Specific Expert Learning and Merging with Auto Geno-GGeee Data, a novel paradigm for eliciting the pre-trained knowledge in T2I models for improved faithfulness based on skill-specific learning and merging of experts. SELMA consists of four stages: (1) collecting skill-specific prompts with in-context learning of LLMs. (2) self-generating image-text samples for diverse skills without the need of human annotation nor feedback from reward models, (3) fine-tuning the expert T2I models on these datasets separately, and (4) obtaining the final model by merging experts of each dataset for efficient adaptation to different skills and mitigation of knowledge conflict in joint training. We illustrate the SELMA pipeline in Fig. 1 (c).
In the first and second stages, we use the LLM and the T21 model generate skill-specific image- text data. The skills include understanding common objects (e.g.. puppy in a backyard). handling long prompts (e.g., an elegant room with floorto-ceiling bookshelves. filled with an impressive collection of books of all genres. The cozy reading nook by the window invites anyone to curl up with a good book.") "), and displaying commonsense-defyin scenes (e.g., cat flying over sky). We aim to teach diverse generation skills to the same T2I model (i.e., self-learning), so that they can handle different types of prompts. To generate image-text pairs for different skills, we first query GPT-3.5 [46] for prompt generation by using only three skill- -kill-specific prompts as in-context examples, and filter the generated prompts with ROUGE-L score to maximize prompt diversity to collect 1K prompts in total (Sec. 3.1). Then we use Stable Diffusion models [59; 50) themselves to generate corresponding images from the prompts (Sec. 3.2). We find that our skill-specific training can help mitigate knowledge conflict when jointly learning multiple skills (see Table 2).
2

---

In the third and fourth stages, we fine-tune a T2I model with the collected image-texi pairs to teach different skills. However, updating the entire model weights can be inefficient; knowledge conflicts within mixed datasets may also lead to suboptimal performance [42]. Thus, in the third stage, we fine- tune T2I models on these self- self-generated image-text pairs with parameter-effiient LoRA (low-rank adaptation) modules [27] to create skill-specific expert T21 models (Sec. 3.3). In the fourth stage, to build joint multi-skill T2I model that can have faithful generations across different skills, we merge the skill-specific experts based on LoRA merging [66; 91] (Sec. 3.4).
We validate the usefulness of SELMA with public -ft------hhhttt T2I models a family of Stable Diffusion - 11.4 [59], v2 [59], and XL [50] on two text faithfulness evaluation benchmarks (DSG [14] and TIFA [28]). three human preference metries (Pick-a-Pie [31]. ImageReward [82], and HPS [79]. and human evaluation. Empirical results demonstrate that SELMA significantly improves T2I models' faithfulness to input text prompts and achieves higher human preference metries. Our final LoRA-Merging model achieves 6.9% improvements on DSG, 2.1% improvements on TIFA, and improves the human preference metries by 0.4 on Pick-a-Pic, 0.39 on ImageReward, and 3.7 on HPS. Furthermore, we empirically show that the T2I models learned from the self-generated images achieve a performance similar to that of learning from ground-truth images (see Fig. 3), Lastly, we further show that fine-tuning with images from weaker T2I model (i.e., SD v2) can help improve the faithfulness of a stronger T2I model (i.e., SDXL), suggesting promising weak-to-sttonn generalization in text-to-image models (see Table 3).
2 Related Work
Training Vision-Language Models with Synthetic Images. As recent denoising diffusion mod- els [69; 25] have achieved photorealistic image synthesis capabilities, many works have studied using their synthetic images for training different models. Azizi el al. [4], Sariyildiz et al. [63], Lei al. [34], inter alia, study training image classification models with synthetic images. For image captioning, Caffagni el al. [9] use diffusion models to generate images on the captioning data. For training CLIP [53] models, several works use diffusion models to generate images from existing captions [74] or text generated with language models [23]. There is a recent research direction using synthetic images to train image generation models themselves, and we discuss more details in the
following paragraph.
Training Text-to-Image Generation Models with Synthetic Images. A line of recent works train text-to-image (T2I) generation models with synthetic images generated by the same or other models annotated with human preference scores using reinforcement learning [33; 82; 79; 20; 17; 21] or direct preference optimization (DPO) [55; 76). While these works show promising results improving model behavior with human preferences, they require expensive human preference annotations. SPIN-Diffusion [87] proposes using self-play [62; 73], which was successfully adopted in Alphago Zero [67] and language models [13; 88), where the model itself becomes judge and iteratively compares itself with previous iterations. However, self-play algorithm still relies on a set of ground truth image-text pairs as positive examples for supervision. Concurrentiindeeedeen to our work, DreamSync [70] trains a T2I model by first creating text prompts with LLMs, sampling multiple images by the T2I model itself, filtering out images with off-the-shelf scorers, and fine-tuning the model on the resulting synthetic image-text pairs [70]. Unlike DreamSyne that depends on image filtering (generating 8 images and taking at most one of them for each text prompt, SELMA generates 1 image for each prompt); significantly improving data generation efficiency by using only 2% of image-text pairs compared with DreamSync. Furthermore, we focus on learning multiple skills with T21 models by learning and merging skill-specific LoRA experts to mitigate knowledge interference across different skills, and we show this approach attains much stronger performance without adding any additional inference cost (see Table 2).
3 SELMA: Learning and Merging Text-to-Image Skill-Specific Experts with Auto-Generated Data
We introduce SELMA, a novel framework to teach different skills to a T21 generation model based on auto-generated data and model merging. As illustrated in Fig. 2, SELMA consists of four stages: (1) skill-specific prompt generation with LLM (Sec. 3.1), (2) image generation with T2I Model (Sec. 3.2), (3) skill-specifer expert learning (Sec. 3.3), and (4) merging expert models (Sec. 3.4).
3

---

Description Examples
Generated Prompts
Skill Prompts
LoRA 1
Prompts
Prampts
LoRA 2
Merge
LoRA
Text-to-lmage
Images
Images
LoRA N
Text Divensity Filter
T21 Model S
T21 Model +
T21 Model +
Final Multi-skil Model
121 Model +
Generated Prompts
Generated Immaees
LoRA
(a) Prompt Generation
(b) Image Generation
(c) Skill-Speelific Expert Learning
(d) Merging Expert Models
Figure 2: Illustration of the four-stage pipeline of SELMA (Sec. 3),
3.1 Automatic Skill-Specific Prompt Generation with LLM
As shown in Fig. 2 (a), we automatically collect skill-specific prompts (that will be paired with images in Sec. 3.2) fine-tune T2I models in two steps: (1) using large language models (LLMs) to generate prompts with brief' skill descriptions and few example prompts and (2) filtering the generated prompts to ensure their diversity. the following, we explain the two steps in detail.
Prompt Generation. We leverage the in-context learning ability of LLMs to generate additional text prompts that follow similar writing styles (e.g., paragraph style) acquire models' knowledge in the same domain (e.g., count capability). We manually collect three seed prompts with similar writing styles or acquire similar skills (e.g., spatial reasoning) the target text prompts. Next, we use these seed prompts as in-context learning examples to query GPT-3.5 (CPT3.5-ttt --ttuuboiinstuuct [46]. We provide additional instructions that encourage diversity in generated prompts, including object occurrences, sentence patterns, and required skills for the T2I model to generate accurate prompts. The detailed prompt template can be found in the Appendix. During prompt generation, keep expanding the seed prompts with the generated prompts, and always randomly sample three prompts as in-context learning examples from the seed prompts.
Prompt Filtering. To improve the diversity of the collected text prompts, we filter out prompts that similar already generated ones. As Taori el al. [72] demonstrate that instruction diversity is crucial for improving the instruction following capability of large language models, we follow the same intuition to create diverse text prompts. To ensure the diversity of generated prompts, first receive a newly generated text prompt from the previous step. Then, we calculate its highest ROUGE- [38] score with all the previously generated and filtered prompts. Following Taori al. [72], we discard text prompts with ROUGE-L>008 to maximize the diversity of generated prompts.
3.2 Automatic Image Generation with Text-to-Image Models
As illustrated in Fig. 2 (b), we generate corresponding images for each generated text prompt using the T2I model. We find that existing diffusion- T2I models are highly effective in learning from their self-generated images, and even benefit from learning from images generated with weaker T2I models (Table 3). It important to leverage the knowledge that already exists inside the T2I models (learned from web data during pre-training), and hence we aim to extract this knowledge for creating the speeifipec image-text pairs, and use them to improve T2I models' faithfulness (Sec. 3.3).
3.3 Fine-tuning with Multiple Skill-Specific LoRA Experts
We efficiently adapt the T2I model different skills by learning skill-speeific Low-Rank Adaptation (LoRA) [27] experts. In LoRA fine-tuning, the updates to the original weights Wo E Rdxd is decomposed with two low-rank matrices: W, AW Wo B.A. where 13 Rdxr, A E Rxxd and r << d. For each new dataset, we fine-tune the T2I model with LoRA independently. and this introduces N skill-specific LoRA experts (as shown in Fig. (c)). In Sec. 5.2, we observe that learning and merging skill-specific experts is more effective than learning single LoRA across all datasets, by helping the T2I model mitigate knowledge conflicts between different skills [42; 11]. However, using multiple skill-specific experts requires the model to know which expert to use for a given input, and this usually requires user annotations on the skill category of inputs. In the next section, we propose to merge skill-specific experts to efficiently construct single multi-skill model.

---

3,4 Merging LoRA Expert Models to Obtain a Multi-Skill Model
Recent work of model merging [29; 68; 71; 83] proposes to merge multiple task-specific weights into one, while retaining the original task-specific performances. Moreover, model merging can help mitigate the knowledge conflicts between datasets because we only need adjust the merging ratios without re- re-training the task-specific models [86; 56]. Due to these benefits, we extend model merging to learn final T2I model that can handle multiple skills without knowledge conflicts. Concretely, given .N LoRA experts learned from Sec. 3.3, we merge all LoRA experts into one N EnEN N A and mergedd
(A merged skills simultaneously (as shown in Fig. 2 (d)). With this approach, we can reach superior performance over standard multi-task LoRA training and even MoF-LoRA (learning a router with LoRA experts), as shown in Tables 2 and 5, and also eliminate the need to know the skill categories beforehand. Note that while ZipLoRA [66] has demonstrated the use LoRA merging (merging 2 LoRA modules) in diffusion models, to the best of our knowledge, we are the first show the effectiveness of LoRA merging on multiple diverse skills (from 5 datasets) in diffusion models.
1 EnEN B");; the resulting single expert can handle all W
4 Experimental Setup
4.1 Evaluation Benchmarks
We evaluate models on two evaluation benchmarks that measure the alignment between text prompts and generated images: DSG [14] and TIFA [28].
DSG consists of 1060 prompts from 10 different sources (160 prompts from TIFA [28], and 100
prompts from each of Localized Narratives [51], DiffusionDB [77]. CountBench [47], Whoops [6], DrawText [41], Midjourney [75], Stanford Paragraph [32], VRD [43], PoseScript [19]). Among the ten DSG prompt sources, we mainly experiment with text prompts from five prompt sources that have (1) ground-truth image-text pairs (to compare the usefulness of auto-generated data with ground-truth data) and (2) measuring different skills required in T21 generation (e.g., following long captions, composing infrequent objects). Specifically, we use COCO [39] for short prompts with common objects in daily life, Localized Narratives [51] for paragraph-style long captions, DiffusionDD [77] for human- numan-written written prompts that specify many attribute details, CountBench [47] for evaluating object counting, and Whoops [6] for commonsense-defying text prompts.
TIFA consists of 4,081 prompts from four sources, including COCO [39] for short prompts with common objects, PartiPrompts [85] / DrawBench [61] for challenging image generation skills, and PaintSkills [15] for compositional visual reasoning skills.
4.2 Evaluation Metries
We quantitatively evaluate the performance of T2I generation models in text faithhulneess and human preference metrics. Specifically, evaluate text faithfulness, we use VQA accuracy from TIFA [28] and DSG [14]. To evaluate human preference score, we use the PickScore [31], ImageReward [82].
and HPS [79] See also Sec. 5.6 for human evaluation. Details can be found in Appendix.
4.3 Implementation Details
In the prompt generation stage (Sec. 3.1), use gpt 1ett--35-turbb-inee instruct [46] to generate text prompts. We collect IK prompts for each of the five datasets (COCO [39], Localized Narratives [51], DiffusionDB [77], CountBench [47], and Whoops [6]). We refer to the resulting auto-generated datasets as Localized SarrativeSELMa SountBench-ELMM.
SSSf WhoopsSfi.Ms and COCOSELMA, and the resulting combination of 5K auto-generated dataset SEGSSI.MA-SK,
In the image generation stage (Sec. 3.2), we use the default denoising steps 50 for all models, and the Classifier-Frre Guidance (CFG) [26] of 7.5. In the LoRA fine-tuning stage (Sec. 3.3), we use 128 as the LoRA rank. During inference, we uniformly merge the specialized LoRA experts into one multi-skill expert (Sec. 3.4). More details can be found in Appendix.
5

---

Table 1: Comparison of SELMA and different text-to-image alignment methods on text faithfulness and human preference (see Sec. 5.1 for discussion). SELMA achieves the best performance in all five metrics when adapted on different base models (i.e., SD v1.4, SD v2, and SDXL). Best scores for each model are bold.
Text Faithfulness
Human Preference DSG prompts
Base Model
Methods
DSG
TIFABUP2
PickScore
ImageRewand f HPS
Base model
67.3
76.6
20.3
-0.22
23.0
(Trainningffme))
SynGen (58)
66.2
76.8
20.4
-0.24
24.5
StructureDiffusion [22]
67.1
76.5
20.3
-0.14
23.5
SD [59]
(RL)
DPOK [2.]]
76.4
-0.26
DDPO [7]
76.7
-0.08
(Automatic data generation)
DreamSyne [70]
77.6
-0.05
SELMA (Ours)
71.3
79.5
20.5
0.36
25.5
Base model
70.3
79.2
20.8
0.17
24.0
SD [59)
SELMA (Ours)
77.7
83.2
21.3
0.72
27.5
Base model
73.3
83.5
21.6
0.70
26.2
SDXL [50]
DyeamSync [70]
85.2
0.84
SELMA (Ours)
80.2
85.6
22.0
Results and Analysis
5.1
Comparison with Different Alignment Methods for Text-to-Image Generation
We compare SELMA with different alignment methods for T2I generation, including training-free methods (SynGen [58], StructureDiffusion [22]), RL-based methods (DPOK [21], DDPO [7]), and DreamSync [70], a concurrent method based on automatic data generation. We experiment with three diffusion-based T2I models (i.e., SD v1.4, SD v2, and SDXL).
SELMA outperforms other alignment methods for T2I generation. As shown Table 1, SELMA consistently improves faithfulness and human preference metries for all three backbones. Specifically, on SD v1.4, SELMA improves the baseline by 2.9 % in TIFA, 4.0% DSG. 0.2 in PickScore, 0.58 in ImageReward, and 2.5 in HPS score. Furthermore, SELMA achieves significantly higher performance than other baselines, including the RL-based methods (DPOK/DDPO), which require annotated human preference data, and DreamSync, concurrent/indeeendeee work based larger auto-geut aellogenerrted dataset (i.e., 28K text prompts; SELMA uses 5K text training prompts in total), and image filtering (i.e., generating 8 images and taking most one of them for each text prompt; SELMA only generates image for each prompt). Besides, on SD v2 and SDXL, SELMA shows larger improvement in text faithfulness (i.e.. 7.4% improvement on DSG for SD v2, and 6.9% on DSG for SDXL), demonstrating the effectiveness of SELMA.
5.2 Effectiveness of Learning & Merging Skill-Specific Experts
We compare (1) separately learning multiple LoRA experts on different auto- auto-generated datasets followed by merging and (2) training single LoRA mixture of datasets. For this, we experi- ment with our five auto-generated image-text pairs: Localized NarrativeSELMA, CountBench DiffusionDBSELMA Whoops and COCOSELMA (see Sec. 4.3 for details).
Learning & merging skill-specific LoRA experts is more effective than single LoRA on multiple datasets. Table 2 shows that the LoRA models trained separately on each of the five automatically generated datasets I. to No.5.) can improve the overall metric over the baseline SD v2 70.3%, while the degree of improvements is different for each metric (e.g.. 76.4% for fine-tuning with Localized NarrativeSELM. and 73.0% for fine-tuning with DiffusionDDSELMA However, training multiple skills simultaneously with a single LoRA (No.6. to No.7.) tends to degrade performance as more datasets are incorporated. This indicates that the T2I model struggles with LoRA to accommodate distinct skills and writing styles from different datasets. A similar phenomenon has been reported VA-MoEE [1 I]. where the knowledge conflict between multiple datasets can degrade the performance. We'r re the first show this knowledge conflict across different skills also

---

Table 2: Comparison of single LoRA and LoRA Merging (see Sec. 5.2 for discussion). We use SD v2 as our base model and train models with our automatically generated image-text pairs. SELMA, auto-geut aeoo-geeerated image-text pairs where prompts are generated with LLMs with three prompt examples from DATA that are not included in DSG test prompts (see Sec. 4.3 for details). LN: Localized Narratives; CB: CountBench; DDB: DiffusionDB. Best/2nd best scores are bolded/underlined
Aute-Generated Training Dataset
Test Faithfulness
Human Preference on USG
No.
Model
LNNELMA
DDB (Rnol Users)
Whoeps
COCO
(Ponggrphy
(Goont)
(Craserr- Facmaa)
Kommuse Objects)
DSG
TIEM
PickSore
ImagsReword
HPS
SDv2
70.3
79.2
20.3
0.17
26.0
76.4
$1.4
20.9
0,.5
26.2
76.0
$1.4
20.3
0.46
25.7
73.0
81.2
20.9
0.45
25.8
Single LoRA
73.0
90.7
20.3
0.64
25.3
25.6
24.8
Merging
/
27.3
exists in diffusion models. We find that merging multiple skill-specific LoRA experts (No.8. and N..9.) achieves the best performance in both text faithfulness and human preference, demonstrating that merging LoRA experts can help mitigate the knowledge conflict between multiple skills.
5.3 Effectiveness of Auto-Generated Data
In this section, we investigate the effectiveness of our automatically generated data by comparing them with ground truth data. We fine-tune SD v2 model using ground truth data from Localized Narratives, CountBench, DiffusionDB, Whoops, and COCO, sampling IK image-text pairs from each dataset and fine-tuning specialized LoRA experts accordingly.
Fine-tuning with auto-generated data can achieve comparable perfor- mance to fine-tuning with ground truth data. As shown in Fig. 3,
GT vs. Auto-Generated Data for Fine-tuning
76.8 77
75.7
76.4
75.875.2
75.1
25.0
we observe that fine-tuning with ei- ther auto-generated or ground truth data improves from baseline SD v2
DSG Accuracy
72.4
73.8
73,0
71.1
18.
performance 70.3%, when evalu- ated on the DSG benchmark. Surpris-
ingly, fine-tuning with the generated data via SELMA outperforms the use of ground truth data in most cases, leading to a DSG accuracy improve- ment of 4.0% % with Localized Narra- tive style prompts, 1.0% % with Count- Bench style prompts, 1.9 % with Dif- fus style prompts, and 0.9% with COCO style prompts. In short, our approach results in an average improvement of 1.2% brought by fine-tuning only auto-generated data without any need for human-collected ground truth text-image pairs, suggesting that diffusion-based text-to-image models may benefit from the diversity of self-generated images. Furthermore, we investigate whether the improvement is brought by text prompt or image quality. We generate images with SD v2 based on 1K ground truth captions, and fine-tune specialized LoRA experts accordingly. We observe that in most cases, using generated images works better than ground truth images (e.g., Localized Narrative), suggesting T2I models can generate images with comparable alignment as ground truth images. Besides, learning from our LLM-generaan with learning from ground truth captions, suggesting the effectiveness of our text prompt collection process. Lastly, we also show in Appendix that fine-tuning with LLaMA3 [1] generated prompts also improve T2I models' faithfulness generation, demonstrating that our proposed SELMA compatible to different LLM-based prompt generator.
SD
50v2 Images
Localized CountBench DiffusionDB
Whoops
coco
Five Datasets Merging)
Narratives
Fine-tuning Datasets
Figure 3: DSG accuracy of SD v2 fine-tuned with different image-text pairs.
captions achieves comparable performance
7

---

Table 3:
Comparison of different image generators for creating training images. In addition to using the same model being trained as an image generator, we also experiment with using a smaller model as an image generator (No. 4.). SDXL is bigger/stronger than SD v2. See Sec. 5,4 for discussion.
Text Falthfulness
Human Preference on DSG
No.
Base Model
Training Image Generator
DSG
TIFA "PPA
PickScore
ImageReward
HPS
1.
SD v2
70.3
79.2
20.8
0.17
24.0
73.3
0.70
26.2
3.
SD
SD
77.7
83.2
21.3
0.72
27.5
4.
SDXL
SD v2
81.3
83.8
21.5
0.78
28.8
80.2
22.0
1.09
29.9
5.4 Weak-to-Strong Generalization
In previous experiments, we demonstrate the interesting self-improving capabilities of T21 models, where the training images were generated by the same T21 model. Here, delve into the following research question: "Can a 721 model benefit from learning with images generated by a weaker model?". The problem of weak-to-strong generalization was initially explored in the context of LLMs [8; 60], referred to as superalignment, which involved training GPT-4 [45] using responses generated by weaker agent, such as GPT-2.
Weaker T2I models can help stronger T2I models. As shown in Table 3, fine-tuning SDXL with generated images from SD v2 (No.4.) remarkably enhances performance over the SDXL baseline (No.2.) in both text faithfulness and human preference. In addition, this approach achieves competitive performance compared with fine-tumn finettunig SDXL with SDXL-generatee images (No.5.), indicating promising potential for weak-to-strong generalization diffusion- T21 generation models. To the best of knowledge, this is the first work find promising improvements in the weak-to-strong generalization for text-to-image diffusion models.
5.5 Comparison with Prompt Generation with LLaMA3
In this section, demonstrate that our proposed paradigm is compat- ible with different prompt genera- tor LLMs. Specifically, we experi- ment with LLaMA3 (8B) [1], pub- licly available open-source LLM and compare the results with GPT-3.5 (gpt-3.5- 1nt--3.-tturbb-instucee setups described in the main paper. With both LLMs, we generate five sets of skill-specific prompts, and each set contains one thousand skill-specific prompts. As shown in Table 4, we find that fine-tuning SDXL with data generated with LLaMA3 achieves 78.6% on average on DSG, improving the baseline by 5.3%, closing the gap GPT-3.5 based results. This demonstrates that SELMA is flexible and compatible with different prompt generator LLMs. Besides, we further experiment with tnn-tuning SDXL with data generated with both weaker image generator SDv2 and weaker prompt generator LLaMA3. Our results show that this model achieves similar perfor- mance as the model fine-tuned with images generated with SDXL, demonstrating that weak-to-strong generalization holds with weaker data generators.
Table 4: DSG and TIFA accuracy of SDXL fine-tuned with prompt data generated with LLaMA3 and GPT-3.5.
Model
Prompt Generator
Image Generator
DSG
SDXL SDXL
73.3
ILaMA3
SDv2
78.0
SDXL
LLaMA3
SDXL
78.6
based
SDXL
GPT3.5
SDv2
81.3
SDXL
GPT3.5
SDXL
80.2
5.6 Human Evaluation
In addition automatic evaluation using text faithfulness benchmarks (DSG and TIFA) and human preference metrics (PickScore, ImageReward, and HPS), we further perform a human evaluation to compare the performance of SDXL and SDXL fine-tuned with SELMA on SEGSSLMA-5K (details in Sec. 4.3). We randomly select 200 prompts from DSG and ask three annotators to determine "Which image aligns with the caption better?" given text prompt and generated images from both SDXL and SDXL+ We provide win/tie/lose options to the annotators, and we report the win vs. lose percentage the following. The user interface, instructions, and the detailed statistics provided in appendix.
8

---

Human Evaluation
Al
67.9%
32.1%
Localized
79.2%
21 8%
Narratives
DiffusionDS
54. 5%
45.5%
CountBench
70. 3%
29.7%
coCO
94.6%
5.4%
SDXL+ (Ours)
Whoops
SDXL
67.7%
32.3%
100
Preference [%]]
Figure 4: Human Evaluation on 200 sampled text prompts from DSG, where we show the win vs.
lose percentages of SDXI. and SDXL+SELMA (Ours).
SDXL+ is preferred than SDXL in terms of text alignment. Fig. 4 shows that all five DSG splits, images generated with SDXL+SELM are preferred 67.9% of the time, compared to 32.1% % for the baseline SDXL. Furthermore, on the five datasets fine-tuned with similar
prompts, SDXL+SELMA achieve a preference rate of 94.6% on COCO split and 79.2% % on Localized Narratives. This substantial preference over the baseline model demonstrates the effectiveness of SELMA in enhancing T2I models.
Table 5: Comparison with different fine-tuning methods on SD v2 with our auto-generated data, in text faithfulness and human preference. See Sec. 5.7 for discussion.
Faithfulness
Human Preference on DSG
No.
Methods
DSG t TIFA
PickScore t ImageReward f HPS
SDv2
70.3
79.2
20.8
0.17
24.0
LoRA Merging (SELMA)
77.7
83.2
21.3
0.72
27.5
LoRA Merging DPO
75.1
81.4
20.8
0.44
26.0
3.
MoF-LLRR
77.2
83.0
21.3
0.68
27.2
5.7 Training Method Ablations
We experiment with various training configurations for SELMA validate our design choices for fine-tuning. As our current experiments are based on supervised fine-tuning with LoRA Merging, we additionally explore Direct Preference Optimization (DPO) [55; 76] as alternative supervised fine-tuning and employing Mixture of Lora Experts (MoE-LoRA) [80] instead of LoRA Merging, See the Appendix for the implementation details. Table 5 demonstrates that while fine-tuning with DPO and MoE-LoRA significantly improves the T2I models' text faithfulness and human preference (No.2 vs. No. simple inference-time LoRA merging achieves the best overall performance. In the end, we adopt LoRA merging and supervised fine-tuning as the default configuration in SELMA for simplicity and efficiency.
5.8 LoRA Expert Size Ablations
In this section, we demonstrate that simply scal- ing the LoRA size doesn't 't help mitigate the knowledge conflict between different skills. We experiment with using single LoRA with dif- ferent ranks (128/ 256 640) and compare them our default LoRA merging (rank=128). Ta- ble 6 shows that increasing the rank of LoRA from 128 to 256 slightly improves the perfor- mance (i.e., 74.9 vs. 74.4), but further scaling rank of the LoRA to 640 significantly drops the performance (i.e., 71.5 vs. 74.4). The performance drop when using LoRA with higher ranks (i.e., rank=640) is similar to the observation Figure [24]. This result indicates the effectiveness of skill-specific learning and merging of LoRA experts.
Table 6: Performance of scaling the LoRA ranks on DSG.
Model
LoRA Rank
DSGMPLIN
Base model (SDv2)
70.3
Single LoRA
128
74.4
Single LoRA
256
74.9
Single LoRA
640
71.5
LoRA Merging
128
77.7

---

SDXL
SEXL+
SDXL
SDXL+SELMA
Acube mede of derim. A cube with the texture denim.
A brown and while cake sitting table. There red and yellow flower petalss around the cake. There while plane the table with fork on top of
A Arain opening its doors. The currently parked train etation. The tran tlue, doors and whiie strips Thare long yellow lne near the train area.
An man iits barbel above his head.
Figure 5: Example images generated with SDXL and SDXLI+ SELMA shows better performance in object composition, attribute binding, and long text prompt following. We highlight the parts of the prompts in red where SDXL makes errors while SDXL+SELMA generates correctly.
5.9 Qualitative Examples
We show some qualitative examples of images generated with SDXL fine-tuned with SELMA paradigm in Fig. 5. We find that fine-tuning with SELMA improves SDXL's capability in composing infrequently co-occurred attributes (i.e., "cube" and "denim" in the top-left image), composing multiple objects mentioned in the text prompts (i.e.. "brown and white cake", "table". "red and yellow flower". and "fork" in the top-right image), following details in the long paragraph-style text prompts (i.e., "blue train with white stripes" and "long yellow line near train area" in the bottom-left image), and generating images that challenge commonsense (i.e., "Old man lifts barbell" in bottom-right image). These qualitative examples demonstrate the effectiveness of SELMA in improving T2I models' text faithfulness and human preference.
6 Conclusion
We propose SELMA, an novel paradigm to improve state-of-the-arr T2I models' faithfulness in generation and human preference by eliciting the pre-trained knowledge T2I models. SELMA first collects self self-generated images given diverse generated text prompts without the need for additional human annotation. Then, SELMA fine-tunes separate LoRA models different datasets and merges them during inference to mitigate knowledge conflict between datasets. SELMA demonstrates strong empirical results in improving T2I models' faithfulness and alignments to human preference and suggests potential weak-to-strong generalization for diffusion- T2I models.
Acknowledgement
We thank the reviewers and Elias Stengel-Eskin Prateek Yadav, and Yushi Hu for the thoughtful discussion. This work supported by DARPA ECOLE Program No. HR001 12390060, NSF-AI Engage Institute 12L-21 DARPA Machine Commonsense (MCS) Grant N66001-19-2-4031. ARO Award W911NF21 10220, ONR Grant 1000014-3-112555 and Bloomberg Data Science Ph.D. Fellowship. The views contained in this article are those of the authors and not of the funding agency.
10

---

References
[1] AI@ Meta: Llama model card https: 20tp::/..t.pp/ghhhhhh..o///eeeee
main/KODEL_ CARD__AAR... md
[2] Ainsworth, S.K., Hayase, J., Srinivasa, S.S.: Git re-basin: Merging models modulo permutation
symmetries. The International Conference on Learning Representations (ICLR) (2023), https: //api. www.eeeeannnncccieeoe.e.e..oeeee
Trg/CorpusIDD 252199400
[3] Ansel, J. Yang, E., He, H., Gimelshein, N., Jain, A., Voznesensky, M., Bao, B., Bell, P., Berard, D., Burovski, E., Chauhan, G., Chourdia, A., Constable, W., Desmaison, A., DeVito, Z. Ellison, E., Feng, W. Gong, J., Gschwind, M., Hirsh, B., Huang, S., Kalambarkar, K., Kirsch, L., Lazos, M., Lezcano, M., Liang, Y., Liang, J., Lu, Y., Luk, C.K., Maher, B., Pan, Y.. Puhrsch, C., Reso. M., Saroufim, M.. Siraichi, M.Y., Suk, H., Zhang. S., Suo, M.. Tillet, P., Zhao, X., Wang. E., Zhou, K., Zou, R., Wang. X., Mathews, A., Wen, W., Chanan, G., Wu, P. Chintala, S.: Pytorch 2: Faster machine learning through dynamic python bytecode trans- formation and graph compilation. In: Proceedings the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2. p. 929-947, ASPLOS '24, Association for Computing Machinery, New York, NY, USA (2024). https: //oii. org/ 10. 1145//6226666555 3640366
https://doi.rrg/100 115/11440655666600000000
[4] Azizi, S., Korbblith, S., Saharia, C.,, Norouzi, M., Fleet, D.J.: Synthetic Data from Diffusion Models Improves ImageNet Classification. TMLR (2023), http://arxix: http:/aawivv.rr.r///////a argg/bs//2e 2204 08466
[5] Betker, J., Goh, G., Jing, L., Brooks, T. Wang, J., Li, L., Ouyang, L., Zhuang, J., Lee, J., Guo, Y., et al.: Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3 pdf 2(3), 8 (2023)
[6] Bitton-Guetta, N., Bitton, Y., Hessel, J., Schmidt, L., Elovici, Y., Stanovsky, G., Schwartz, R.: Breaking common sense: Whoops! a siiion-and-languaal benchmark synthetic and compositional images. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 2616-2627 (2023)
[7] Black, K., Janner, M., Du, Y., Kostrikov, I., Levine, S.: Training diffusion models with reinforcement learning. The International Conference Learning Representations (ICLR) (2024)
[8] Burns, C., Izmailov, P., Kirchner, J.H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., et al.: Weak Weak-to-streee to-onn generalization: Eliciting strong capabilities with weak supervision. arXiv preprint 132.v:2311209990000 (2023)
[9] Caffagni, D., Barraco, M., Cornia, M., Baraldi, L., Cucchiara, R.: Synthcap: Augmenting transformers with synthetic data for image captioning. In: Foresti, G.L., Fusiello, A., Hancock, E. (eds.) Image Analysis and Processing ICIAP 2023. pp. 112-123. Springer Nature Switzerland, Cham (2023)
[10] Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L.. Yang, M.H., Murphy, K., Freeman, W.T., Rubinstein, M., et al.: Muse: Text-to-image generation via masked generative transformers. ICML (2023)
[11] Chen, S., Jie, Z., Ma, L.: Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milms. arXiv preprint arXiv:2401.15947 (2024)
[12] Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.: Pali: A jointly-scaled multilingual language-image model. ICLR (2023)
[13] Chen, Z., Deng, Y., Yuan, H., Ji, K., Gu, Q.: Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024), http: /ttp::////xii...//e org/ www./r/g//////eeeeeee/eeeee abs/2401 2tt.//441111111.3
[14] Cho, J., Hu, Y., Garg, R., Anderson, P., Krishna, R., Baldridge, J., Bansal, M., Pont-Tuset, J., Wang, S.: Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-image generation. ICLR (2024)

---

[15] Cho, J. Zala, A., Bansal, M.: Dall-eval: Probing the reasoning skills and social biases of -itate generation models. In: ICCV (2023)
[16] Cho, J., Zala, A., Bansal, M.: Visual programming for text-to-image generation and evaluation. In: NeurlPS (2023)
[17] Clark, K., Vicol, P., Swersky, K., Fleet, DJJ.: Directly Fine-Tuning Diffusion Models
Differentiable Rewards. In: ICLR (2024), http:/ /ttp/aixvvvorr// /209://233..0. 17400
[18] Dai, X., Hou, J., Ma, C.Y., Tsai, S., Wang, J., Wang. R., Zhang, P., Vandenhende, S., Wang, X., Dubey, A., et al.: Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint arXiv:2309.1588 (2023)
[19] Delmas, G., Weinzaepfel, P., Lucas, T., Moreno-Noguer, F., Rogez, G.: Posescript: 3d human poses from natural language. In: European Conference on Computer Vision. pp. 346-362.
Springer (2022)
[20] Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., Zhang, T.: RAFT: Reward rAnked FineTTuning for Generative Foundation Model Alignment. TMLR (2023), http://arxix org/ abs wwwpa304.cae 2220/2334.0007 06767
[21] Fan, Y., Warkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C.,, Abbeel, P., Ghavamzadeh, M., Lee, K., Lee. K.: Dpok: Reinforcement learning for fine-tuning text-to -onateo-mmae diffusion models. In: NeurIPS (2023)
[22] Feng. W., He, X., Fu, T.J., Jampani, V., Akula, A., Narayana, P., Basu, S., Wang, X.E., Wang. W.Y.: Training-free structured diffusion guidance for compositional text-to-image synthesis. ICLR (2023)
[23] Hammoud, H.A.A.K., Itani, H., Pizzati, F., Torr, P., Bibi, A., Ghanem, B.: SymthCLIP: Are We Ready for Fully Synthetic CLIP Training? (2024), attp:///rxix: http://wwwwv............... argg/bss//022.. 01832
[24] He, H., Li, J.B., Jiang, X., Miller, H.: Sparse matrix in large language model fine-tuning. arXiv preprint arXiv:2405. 15525 (2024)
[25] Ho, J., Jain, A.. Abbeel, P.: Denoising Diffusion Probabilistic Models. In: NeurIPS, pp. 1-25 (2020), http::///orr...oorraa////006 11239
[26] Ho, J., Salimans, T::
Classifier-fres diffusion guidance. In: NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications (2021), https:://ppenreview net/
forum?id-qu8AKssEYtI
[27] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. ICLR (2022)
[28] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., Smith, N.A.: Tifa: Accurate and interpretable text-to- text-to-image faithfulness evaluation with question answering. ICCV (2023)
[29] Ilharco, G., Ribeiro, M.T., Wortsman, M., Gururangan, S., Schmidt, I.., Hajishirzi, H., Farhadi, A.: Editing models with task arithmetic. The International Conference on Learning Representa- tions (ICLR) (2023), https: //tp//appp./ semanticecholarr waaeeroor//oCoee Inng/CorpusID:::0 25444408499
[30] Kang, M., Zhu, J.Y., Zhang. R.. Park, J., Shechtman, E.,, Paris, S., Park, T.: Scaling up gans for text-to-image synthesis. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR) pp. 20124-10134 (2023), https://apl semanticscholl org hetg/CorpussD. 257427461
[31] Kirstain, Y.. Polyak. A., Singer, U., Matiana, S., Penna, J., Levy, O.: Pick-a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems 36 (2023)
[32] Krause, J., Johnson, J., Krishna, R., Fei-Fei, L.: A hierarchical approach for generating descriptive image paragraphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 317-325 (2017)
12

---

[33] Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Gu, S.S.: Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192 (2023)
[34] Lei, S., Chen, H., Zhang, S., Zhao, B., Tao, D.: Image Captions are Natural Prompts for Text-to-lmage Models (2023), http://arxivy org/ abs/2307 08526
[35] Li, C., Xu, H., Tian, J., Wang, W., Yan, M., Bi, B., Ye, J., Chen, H., Xu, G., Cao, Z., et al.: mplug: Effective and efficient vision-language learning by cross moos- skip-connections, ACL (2022)
[36] Li, J., Li, D.,, Xiong, C., Hoi, S.: Blip: Bootstrapping engguage-image pre- pre-training for uni- fied vasion-language understanding and generation. In: International Conference on Machine Learning. pp. 1888-119000 PMLR (2022)
[37] Li, Y., Liu, H., Wu, Q., Mu, F., Yang. J., Gao, J., Li, C.,, Lee, YJ.:: Gligen: Open-set grounded snt-tt--mmage generation. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22511 1-2221 (2023)
[38] Lin, C.:.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74-81 (2004)
[39] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[40] Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B.: Compositional visual generation with composable diffusion models. In: European Conference on Computer Vision. pp. 423-439. Springer (2022)
[41] Liu, R., Garrette, D.,, Saharia, C., Chan, W., Roberts, A., Narang, S., Blok, I., Mical, R., Norouzi, M., Constant, N.: Character-aware models improve visual text rendering. ACL (2023)
[42] Liu, S., Liang. Y., Gitter, A.: Loss-balanced task weighting to reduce negative transfer in
multi-task learning. In: AAAI Conference on Artificial Intelligence (AAAI) (2019), https:
/ happ..sematiischoo....
[43] Lu, C., Krishna, R., Bernstein, M., Fei-Fei, L.: Visual relationshipp detection with language priors. In: Computer Vision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings. Part 14. pp. 852-869. Springer (2016)
[44] maintainers, T., contributors: Torchvision: Pytorch's 's computer vision library. httpayyytttss///ihhttt com com/pytoocc/vvision (2016)
[45] OpenAl: Gpt-4 technical report (2023), https://api http::/ppp....attttccccooorrrrrs 257532815
[46] OpenAl: Openai models (2023),
[47] Paiss. R.. Ephrat, A., Tov, O., Zada, S., Mosseri, I., Irani, M., Dekel, T.: Teaching clip to count ten. ICCV (2023)
[48] Phung, Q. Ge, S., Huang, J.B.: Grounded text-to- -ot-toe-imggee synthesis with attention refocusing. arXiv preprint arXiv:2306.05422 (2023)
[49] von Platen, P., Patil. S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj. M., Wolf, T.: Diffusers: State-of-the-art diffusion models. https://githuii com/huggingface/ diffusers (2022)
[50] Podell, D., English, Z.,, Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. ICLR (2024)
13

---

[51] Pont-Tuset, J., Uijlings. J., Changpinyo. S., Soricut, R., Ferrari, V.: Connecting vision and lan-
guage with localized narratives. In: Computer Ecdin-ECCV 2020: 16th European Conference. Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. pp. 647-664. Springer (2020)
[52] Prabhudesai. M., Goyal, A., Pathak, D., Fragkiadaki, K.: Aligning text-to- -omag diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739 (2023)
[53] Radford, A., Kim. J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P.. Clark, J., Krueger, G., Sutskever, I., Wook, J., Chris, K., Aditya, H., Gabriel, R., Sandhini, G., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I:: Learning Transferable Visual Models From Natural Language Supervision. In: ICML (2021),
http: / wwwpv..ivvvvv/////v///a////aaava org/
00020
[54] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748-878887 8763. PMLR (2021)
[55] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct Preference Optimization: Your Language Model is Secretly Reward Model. In: NeurIPS (2023), http: wwwpxiivrrg/r//////ee/
//arxiv. org/ abs abs/abb5.555. 2305. 18290
[56) Ramé, A., Couairon, G., Shukor, M., Dancette, C., Gaya, J.B., Soulier, L.,, Cord, M.: Re- warded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Conference on Neural Information Processing Systems (NeurIPS) (2023), attps://ap...eenatiicchooaa
semanticscholar. www.laa..rrrrrppooeerree :ttp//oppuD:::5599611..
[57] Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-image generation. In: International Conference on Machine Learning. pp. 8821-8831. PMLR (2021)
[58] Rassin, R., Hirsch, E., Glickman, D., Ravfogel, S., Goldberg, Y., Chechik, G.: Linguistic binding in diffusion models: Enhancing attribute correspondence through attention map alignment. Advances in Neural Information Processing Systems 36 (2024)
[59] Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 10684-10695 (2022)
[60] Saha, S., Hase, P., Bansal, M.: Can language models teach weaker agents? teacher explanations improve students via personalization. In: Advances in neural information processing systems (NeurIPS) (2023)
[61] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, K., Gon- tijo Lopes, R., Karagol Ayan, B., Salimans, T., et al.: Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems 35, 30433644796666 36494 (2022)
[62] Samuel, A.L.: Some studies in machine learning using the game of checkers. IBM Journal of Research and Development 3(3), 210-229 (1959). https://doi.org/10.. 147///.....3 010
[63] Sariyildiz, M.B., Alahari, K., Larlus, D., Kalantidis, Y.: Fake it Till You Make it: Learn- ing Transferable Representations from Synthetic ImageNet Clones. In: CVPR (2023).
[64] Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarezyk, R., Mullis, C., Jitsev, J., Komatsuzaki, A.: LAION-400M: Open dataset of CLIP-filterd 400 million image-text pairs. In: Proceedings of Neurips Data-Centric AI Workshop (2021)
[65] Segalis, E., Valevski, D., Lumen, D., Matias, Y., Leviathan, Y.: A picture is worth a thousand words: Principled recaptioning improves image generation. arXiv preprint arXiv:2310. 16656 (2023)
14

---

[66] Shah, V., Ruiz, N., Cole, F., Lu, E., Lazebnik, S., Li, Y., Jampani, V:: Ziplora: Any subject in any style by effectively merging loras. ArXiv abs/2311.13600 (2023), https: //api. seapp.ssemanicsccho..e.
Org/CorpusIDD-... 26:33551666
[67] Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L.., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., Hassabis, D.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587),
484-489 (jan 2016). https:/doi.org//00 1033/aatrtt/teeeeeeeeeeeee 16961
[68] Singh, S.P., Jaggi, M.: Model fusion via optimal transport. Conference on Neural Information Processing Systems (NeurIPS) (2020), https:/ //api. semanticscholar org/CorpusID, 204512191
[69] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. In: ICML (2015)
[70] Sun, J., Fu, D., Hu, Y., Wang. S., Rassin, R., Juan, D.C., Alon, D., Herrmann, C., van Steenkiste, S., Krishna, R., et al.: Dreamsyne: Aligning text-to-image generation with image understanding feedback. arXiv preprint urXiv:2311.1793 (2023)
[71] Sung, Y.L., Li, L., Lin, K., Gan, Z., Bansal, M., Wang, L.: An empirical study of multimodal model merging. Empirical Methods in Natural Language Processing (Findings) (2023)
[72] Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., Hashimoto, T.B.: Stanford alpaca: An instruction-following llama model. https://githun com totmm/ttsu-laab cranford_aapaca (2023)
[73] Tesauro, G.: Temporal difference learning and td-gammon. Commun. ACM 38(3), 58-68 (mar 1995). http:://doo.org/10.111522255 203343
https://doi. ortp::///ooioor/1..... 11211..111//230333 203330.
[74] Tian, Y., Fan, L., Isola, P., Chang, H. Krishnan, D.: StableRep: Synthetic Images from Texl-to-Image Models Make Strong Visual Representation Learners. In: NeurIPS (2023), http://arrt wwwpoor//////o/oooo/oe 20982430.00984
[75] Turc, I., Nemade, G.: Midjourney user prompts generated images (250k) (2023)
[76] Wallace, B., Dang. M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam. S., Ermon, S., Xiong, C., Joty, S., Naik, N.: Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:231 12908 (2023)
[77] Wang. Z.J., Montoya, E., Munechika, D., Yang, H., Hoover, B., Chau, D.H.: Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. ACL (2023)
[78] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T. Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T.L.,, Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of- the-art natural language processing. In: Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38-45. Association for Computational Linguistics, Online (Oct 2020), httpps://wwtt aclweb. org/ anthology/ 2020. munlp-demos.
[79] Wu, X., Sun, K., Zhu, F., Zhao, R., Li, H.: Human Preference Score: Bet- ter Aligning Text-to-image Models with Human Preference. In: ICCV (2023). https://doi.orr/10.. wtt.11100/cevv1000001... 00200,
ab////b/////.///////// 14420
[80] Wu, X., Huang, S., Wei, F.: Mole: Mixture of lora experts. In: The Twelfth International Conference on Learning Representations (2023)
[81] Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, image synthesis with training-free box-constrained diffusion. Proceedings of the IEEF/CVF International Conference on Computer Vision. pp. 7452-7461 (2023)
W., Zheng, Y., Shou, M.Z.: Boxdiff: Text-to-
15

---

[82] Xu, J., Liu, X., Wu, Y. Tong, Y., Li, Q.. Ding, M., Tang, J., Dong, Y: Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS (2023)
[83] Yadav, P., Tam, D., Choshen, L., Raffel, C., Bansal, M.: Ties-merging: Resolving interference when merging models. In: Advances in Neural Information Processing Systems (NeurIPS) (2023)
[84] Yang. Z., Wang. J., Gan, Z., Li, L., Lin, K., Wu, C., Duan, N., Liu, Z., Liu, C.,, Zeng, M., al.: Reco: Region-controlled text-to-image generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14246-14255 (2023)
[85] Yu, J., Xu, Y., Koh, J.Y., Luong, T., Baid, G., Wang, Z., Vasudevan, V., Ku, A.. Yang, Y., Ayan, B.K., al.: Scaling autoregressive models for content-rich text-to-image generation. TMLR 2(3), 5 (2023)
[86] Yu, I.., Bowen, Y., Yu, H., Huang, F., Li, Y.: Language models are super mario: Absorbing abilities from homologous models as a free lunch. ArXiv abs/2311.03099 (2023), https:
/ //api. saap..ssmmnticscollrr
Org/Corpus 265034087
[87] Yuan, H.. Chen, Z., Ji, K., Gu, Q.: Self-Play - of Diffusion Models for Text-to-Image Generation (2024), http: /tap:///axivvi http://aaxxii.oogaaa//////////// wtggg/ab//442..10020 10210
[88] Yuan, W., Pang, R.Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., Weston, J.: Self-Rewarding Language Models (2024), http: /ar/aaxxvvvv.... //tp://i.i....r/r//////1111/ 10020
[89] Zhang. L., Rao, A., Agrawala, M.: Adding conditional control text-to-image diffusion models. In: Proceedings of the IEEE/CVF Internationa Conference on Computer Vision. pp. 3836-3847 (2023)
[90] Zhang, Y., Jiang, L., Turk, G., Yang. D.: Auditing gender presentation differences in text-to- image models. arXiv preprint arXiv:2302.03675 (2023)
[91] Zhong, M.. Shen, Y., Wang. S., Lu, Y., Jiao, Y. Ouyang, S., Yu. D., Han, J.. Chen, W.: Multi-lora composition for image generation. arXiv preprint arXiv:2402. 16843 (2024)
16

---

Appendix
In this appendix, we present the following:
Details of evaluation metrics we use (Appendix A).
Evaluation on HPS v2.1 benchmark (Appendix B).
Additional qualitative examples with SELMA on SDXL backbone (Appendix C).
Spill-specific VQA accuracy both TIFA and DSG benchmarks (Appendix D).
Human evaluation details (Appendix E).
Additional comparison with fine-tuning with filtered generated data (Appendix F).
Bias evaluation in fine-tuned T2I models (Appendix G).
Implementation details of SELMA (Appendix H).
Implementation details of two training configuration variants: DPO and MoE-LoRA (Ap- pendix I).
Prompts we used to query LLM to generate new text data (Appendix J).
Limitations and broader impact of SELMA approach (Appendix K).
License information for data and model used in this paper (Appendix L..
A Evaluation Metrics
We quantitatively evaluate the performance of T2I generation models in text faithfulness and human preference metrics.
To evaluate T2I model's faithfulness in generation, we use VQA accuracy from TIFA and DSG. Specifically, TIFA and DSG utilize LLMs to generate questions given a text prompt and utilize the VQA model check whether can answer the questions correctly given the generated image. The image is considered to have better faithfulness to text prompts if the VQA model can answer the question more correctly. For TIFA, we use BLIP-2 as the VQA model following Sun al. [70], For DSG, we use mPLUG-large [35] as the VQA model, PaL.I [12] is not publicly accecssible, and Hu el al. [28] shows that mPLUG achieves higher human correlation than BLIP-22.
Text faithfulness.
Human preference metrics. To evaluate how the generated images align with human preference, we the PickScore [31], ImageReward [82], and HPS [79]. PickScore and HPS are based on CLIP [54] trained on the Pick-a-Pic dataset [31] and Human Preference Score dataset [79] respectively, which both have annotations of human preference over images. ImageReward
BLIP [36] based reward
model fine-tuned on human preference data collected on DiffusionDB. We calculate PickScore, ImageReward, and HPS on the 1060 DSG prompts. We also provide the evaluation results on HPS prompts in the appendix.
B Evaluation on HPS v2.1 Benchmark
In the main paper, we caleulate HPS score [79] text prompts DSG benchmark, following DreamSync [70]. In this section, we additionally show the HPS score on the prompts from HPS v2.1 benchmark. HPS benchmark contains 3200 unique prompts from four different categories: anime, concept-art, paintings, and photo. We calculate the HPS score based on its HPS v2.1 model trained higher quality datasets. As shown in Table 7, when adapting SELMA to different stable diffusion base model, our approach significantly improves the baseline performance (i.e., 2.5 for SD v1.4, 4.3 for SD v2, and 1.4 for SDXL), achieving better performance than all the released model the HPS benchmark.
BBnchmark performance can be found: https: //eg:://gttttt http:p///ggtttl...mmmg/ss///////
17

---

SDXL
SEXL+
SDXL
SDXL
Counting
A carloon vedoo ilustralion sel of four cule bumble bee.
Seven groomemen the cereroory..
Text Rendering
Fall here
FALL HER
KNOWEDCE POWER
Studio shot of whes in the shape text knowledge power sprouting, contered.
'Fall is here' written in autumn leaves flceting on lake.
Entity
cchid draws blue flower pencil.
road indicates dinesuur
Spatial Relationship
STOP
we dog running. hoiding a
dog stop sign; stop sign dog.
mouth and the background is not clearly visible.
Attribute
Hudson river school ice cream landscape chocolate river mountains candy
surting a lorge wave.
Figure Qualitative example images generated with SDXI. and SELMA (Ours) from DSG [14] test prompts requiring different skills. SELMA helps improve SDXL in various skills, including counting, text rendering, spatial relationships, and attribute binding. highlight the parts of the prompts in red where SDXL makes errors while SDXL+SELMA generates correctly.
18

---

Table 7: Evaluation on HPS v2.1 evaluation benchmark. SELMA achieves signifiantly better scores on HPS evaluation benchmark compared with baselines, and outperformin all other baselines reported in HPS v2.1 benchmark.
HPS v2.1 Evaluation Benchmark
Method
Anime
Concept-Art
Paintings
Photo
Average
SD v1.4 [59]
26.0
24.9
24.8
25.7
25.4
+ SELMA
28.2
27.6
27.8
28.0
27.9
SD v2 [59]
27.1
26.0
25.7
26.7
26.4
+ SELMA
32.0
30.3
30.0
30.4
30.7
SDXL [50]
33.3
32.1
31.6
28.4
31.3
+ SELMA
34.7
32.7
32.6
30.8
32.7
Table 8: Detailed skill-specific comparison of SD models vs. SD MOLLl on TIFA bench-
mark.
Method
TIFA skills
Animal/Hommm
Object
Location
Activity
Color
Sputial
Attributc
Food
Counting
Material
Other
Shape
Averge
SD SDv11.1550 (59)
80.3
76.6
SELMA
87.1
8488
75.9
74.4
76.01
38.1
66.2
78.5
52.2
59.4
79.5
SD v2 (59)
86.5
82.6
83.8
75.6
76.8
62.4
75.4
85.2
66.5
82.2
55.4
75.0
79.2
SELMA
89.7
88.0
$7.6
80.3
80.3
66.0
77.2
91.0
65.8
81.3
63.2
68.1
83.2
SDXL SELMA
90.3
86.4
86.6
80.0
78.6
67.7
78.3
90.6
67.4
84.2
67.7
62.3
82.9
C Additional Qualitative Exmaples
In Fig. 6, we show additional qualitative examples of SDXL and SDXL+SELMA from DSG [14] test prompts requiring different skills. SELMA helps improve SDXL in various skills, including counting. text rendering, spatial relationships, and attribute binding. For counting skill prompts, SDXL+SELMA generates "four bees" and "seven groomsmen" correctly following the text prompts. For text rendering skill prompts, SDXL+SELMA can render the text ("knowledge power" and "Fall is here") more accurately, while still lacks the capability to render the text in the texture of vines or autumn leaves. For entity skill (placing correct objects) prompts, the SDXL sometimes misses some entities mentioned in the text prompt (i.e.. "A child", and "A road sign"), while SDXL+SELMA can successfully generate them. For spatial relationship skill prompts, SDXL+SELMA generated images (i.e., "holding stick in its mouth", and "stop sign left to dog"). Lastly, for attribute skill prompts, SLLL+SELMA binds objects with their corresponding attributes (ie.. "cotton candy trees" and "A man in black suit") more accurately than SDXL. These qualitative results demonstrate the effectiveness of SELMA.
D Skill-specific VQA Accuracy on TIFA and DSG
In this section, we show the detailed VQA accuracy for each skill category on TIFA and DSG benchmarks. Since DreamSync [70] does not provide the skill-specific scores, we report the skill- specific scores of SD models on TIFA and DSG and based on our experiments in Table 8 and Table 9; we observe there are less than 1% score differences of SD/SDXL models TIFA average accuracy between the results in Sun et al. and ours.
As shown in Table and Table 9, on both TIFA and DSG benchmarks, SELMA improves the generation faithfulness in most of the categories. Comparing SDXL and SDXL+SELMA, the SDXL finetuned with SELMA approach shows large improvement especially in entity (i.e., 3.1% on animal/human on TIFA compared with SDXL, 5.6% in whole on DSG, 12.2% in part on DSG), as well as spatial relationship (i.e., 1.9% on TIFA, and 8.0% on DSG), and counting skills (i.e. 1.4% on TIFA, and 13.2% on DSG). Besides, we also observe that SELMA significantly improves the text rendering for SD v2 and SDXL. (i.e., 16.4% compared with SDXL,, and 6.5% compared with SD v2 on DSG), but not SD v.4 (i.e., 1.8% decrease compared with SD v1.4 on DSG).
19

---

Table 9: Detailed skill-specific comparison of SD models vs. SD models+SELMA on DSG bench- mark. We show the skill categories that have more than 50 questions.
Method
Whole
Color
Shape
Sputial
Part
State
Count
Action
Global
Material
Type
Text Rendering
Average
62.5
58.2
52.7
+ SELMA
83.7
62.5
52.0
66.2
72.1
63.3
66.1
72.1
57.1
59.7
67.5
50.9
71.3
SD v2 159]
80.8
68.6
50.0
63.6
72.3
63.6
62.9
61.9
55.7
66.8
60.9
70.3
SELMA
88.0
80.8
65.4
71.0
78.7
71.5
66.3
78.4
61.0
69.2
81.4
67.4
77.7
SDXL 150]
84.5
74.7
58.0
69.4
71.1
60.7
59.8
71.7
61.5
63.9
71.7
60.0
73.3
Human Evalcation
ONE WAY
Dooluninn 14 200
Caption photo of ongn with 'one way
Wh ch Image gigsa with the caction better?
Imaget
Tie
Figure Example user interface for human evaluation on DSG prompts.
E
Human Evaluation Details
Table 10: Human Evaluation 200 sampled text prompts from DSG. We show the detailed win/lose/tie counts on all samples and samples from each dataset.
We conduct the human evaluation (described in the main paper See. 5.5) on 200 randomly sampled prompts from DSG, with three external annotators. We show the annota- tion interface Fig. 7. The image order between the two models is randomized avoid the leakage of information about which image is generated with which model.
Eval Dataset
Win
Lose
Tie
All
241
114
245
In Table 10, we show the detailed annotator votes for win, lose, and tie for SDXL and SDXL+SELMA. SDXL+SELMA has significantly higher win votes com- pared with SDXL on all the 200 sampled text prompts (i.e., 241 win vs. 114 lose), demonstrating the effectiveness of SELMA.
Localized Narratives DiffusionDB CountBench COCO Whoops
19
5
12
5
34
26
11
17
35
2
47
21
10
26
F Comparison with Fine-tuning with Filtered Generated Data
We demonstrate that our automatically collected data is of high quality to improve the faithfulness of T21 models. Specifically, we filter out the generated images based on TIFA score (>0.9) and VILA score (>0.6), and maintain IK data for each set of skill-specific prompts. Fine-tuning with
20

---

filtered images achieves 80.3% DSG score with SDXL backbone, which is similar performance as fine-tuning with un-filtered version (i.e., 80.2% on DSG), indicating the overall high quality of the data, which does not lead to performance degradation.
G Bias Evaluation in T2I Models
In this section, we investigate whether fine-tuning with T21-generated images will increase the bias in existing T2I models. Specifically, we evaluate the T2I model with GEP score [90], which measures the gender bias in T21 models. Specifically, learned cross-modal classifier is used to distinguish attributes (e.g., dress) in the image, and GEP score calculates attribute-wisc difference based on output score from the cross-modal classifier. A lower GEP score indicates less gender bias. Fine-tuning SD v2 with our generated data achieves 0.04413 GEP score, while fine-tuning with ground truth data achieves 0.04976. This indicates that tun-ttning with our generated data does not amplify the bias in existing T21 models compared with fine-tuning with ground truth data.
H Implementation Details of SELMA
In the prompt generation stage (Sec. 3.1), we use -....-ttrrb---sstrccc [46] to generate text prompts by providing three prompts for each skill as in-context examples. For each of the five datasets (COCO [39]. Localized Narratives [51], DiffusionDB [77], CountBench [47], and Whoops [6]), we collect IK prompts starting with three prompts randomly sampled from them, ensuring the prompts are not included in the DSG test prompts (i.e., 5K prompts in total). We refer to the resulting auto-generated datasets as Localized Narrative CountBench SELMA, Whoops DiffusionDB-SELM... auto-generater dataset as DSGSELMA.SKK
in
and COCOSELMA, We refer the resulting combination of 5K
In the image generation stage (Sec. 3.2), we the default denoising steps 50 for all models, and the Classifier-Free Guidance (CFG) [26] of 7.5. In the LoRA fine-tuning stage (Sec. 3.3), we use 128 as the LoRA rank. We fine-tune LoRA in mixed precision (i.e., FP16) with constant learning rate of 3e-4 and a batch size of 64. We fine-tune LoRA modules for 5000 steps, which is approximately 313 epochs. During inference, expert (Sec. 3.4). We evaluate model checkpoints every 1000 steps and pick the model with the best text faithfulness on DSG benchmark. Fine-tuning LoRA for SD v1.4, SD v2, and SDXL takes 6 hours, 6 hours, and 12 hours on single NVIDIA L40 GPU, respectively. We use Diffusers [49] for our experiments.
uniformly merge the specialized LoRA experts into one multi-skill
I Implementation Details of DPO and MoE-LoRA
In this section, we provide the implementation details of two training approaches we experiment with (described in Sec. 5.6 in the main paper).
Direct Preference Optimization (DPO). We fine-tune LoRA models with DPO proposed in [76]. Specifically, we sample two images with T21 models and calculate the image-text alignment with CLIP score [54]. We use the image with a higher CLIP score as the positive example and the image with lower CLIP score the negative example. We fine-tune the LoRA models to learn to generate images closer to the positive image distribution and push away from generating images similar to the negative image distribution. Similarly, we fine-tune with DPO on five datasets with different text styles and skills and merge LoRA expert models during inference time by averaging the LoRA weights. In DPO training, we constant learning rate 3e-4 and fine-tune LoRA for 5K steps. We evaluate the model on DSG every IK steps and pick the best checkpoint.
Mixture of Lora Experts (MoE-LoRA). MoE-LoRA [80] utilizes a gating function (router) to decide which experts to use during training and inference. The gating function predicts weights for each expert based on layer inputs and picks the top K experts to use at each layer. Specifically. the gating function we use is a simple linear mapping function, where [
Wz,. x is the input to each layer, W, is the learnable gating weights, and {w, [ are the predicted weights. The outputs of each expert are added together with the normalized weights from the gating function. In MoF-LLRA, A, we initialize five LoRA experts fine-tuned on different datasets containing different text styles and skills. We freeze the learned LoRA weights and only fine-tune the gating function
21

---

on the collected five datasets. We also compare with learning LoRA weights along with the router, which achieves worse performance (75.9 on DSG). We activate all five experts during training and inference (i.e.. K = 5). In MoF-LoRA training, we use constant learning rate 1e-5 and fine-tune LoRA for 5K steps. We evaluate the model on DSG every 1K steps and pick the model with highest text faithfulness score.
Narratives:
mage desorphas plons.
creating image
descrition
The descriptinns mention attiiuute image generator: spatisl,
tmage
life
The desomptions should be
The text
descrittins should different from each other. but generaty paragraph exprions.
Herchr some image description examples:
CountBench:
Please 20
imaga
crealing image
The descriptions should evaluate modal's capability by
be merrion diverse objects that frequentt rppoorr daily life
The image description
The image descripiion should be lox then 30 words
The leet style the discriptions should be difierent from wach other.
Harehe some image description examples.
DiffusionDB:
Please create 20 desception:
several
the
The image should be used to evaluste one of the skills of Image genorator: spatial, count,
animal, human, object location, astivity, color, atribute, food, material shape. The image desoritions need menton diverse objects that froquenty appear in daily life.
The image desoritons should be less than 50 words. The text style of the mage descriptions should be different from each other, but descr be the image
in details.
Horo're some image desoription examples:
coCo:
Phease crudta 20 more diverss descr Here seversl rule for crieting the description
The imag ahould be used to svaluate one the skills of image generstor: spelia,, count, anamal human, object. location. adtivty, color, atribute, food. material shape.
3. The imae desoritins need menton diverse otjects that frequenty sppear in daily life
The smage descrptions should be teox than 30 words
5. The tucl style the mage descriptions ahould be difirrent froms each other
Here're some image desortation examples:
Whoops:
Please create 20 descmption:
several rules the
skills arimm: humenn object, location, adtilly color, atrituuee food, material shopw.
The should used wvskuste
generator: count,
3. The image desoritions need menton diverse objects that froquenty appear in daily life. but have some commensense defying combination
The image desorptons should be less than 30 words.
5. The text style the mage description should be difereet from each other.
Herehs some image diseription ecomples:
Figure 8: Prompts used to query GPT-3.5 auto auto-generate new prompts targeting different skills.
Skill-Specific Prompt Generation Details
We show the prompts we use to query GPT3.5 to generate 1K prompts for each skill. As shown in Fig. 8, we use different prompts to generate SELMA data. For example, specify "paragraph captions" to generate text prompts that can be used to teach model to follow long text prompts, and specifying "evaluate model's capability to count objects" to collect set of prompts for improving model's 's counting capability. Besides, all the prompt generation, we emphasize that the image should "mention diverse objects" to maximize the semantic diversity in generated prompts.
22

---

K Limitations and Broader Impact
Text-to-image generation models can be used in many real-world applications, such as creating content for media and entertainment. Our proposed SELMA improves T21 models' faithfulness with auto-generated data, reducing human annotation efforts for high-quality image-text pairs.
However, we also note that our SELMA has several limitations. SELMA relies on a strong image generator and inslooction-followinng LLM. Note that SELMA is model-agnostic and can be implemented with publicly accessible models (GPT-3.5LLLaMAS and Stable Diffusion models). Also, since our fine-tuning works well with small number of image-text pairs (i.e., for each skill, only generate IK text prompts and generating one image each text prompt), the cost of LLM inference (i.e., $27.78 for querying GPT-3.5 for generating prompts in all the experiments including ablation studies) and image generation (8s per image for image generation with SDXL on a single NVIDIA L40 GPU with 48GB memory) is minimal. Besides, although SELMA helps boost T2I models' performance significantly in both text faithfulness and alignment to human preference, fine-tuning with SELMA does not guarantee the resulting model follow the text prompts in every detail. To use T2I models trained with SELMA, researchers should first carefully study their capabilities in relation to specific context in which they are being applied.
L Licenses
We provide the licenses of the existing assets we use in this paper in Table 11.
Table 11: A list of the licenses of the existing assets used in this paper.
Asset
License
PyTorch [3]
BSD-style
Huggingface Transformers [78]
Apache License 2.0
Torchvision [44]
BSD 3-Clause "New" or "Revised" License
Diffusers [49]
Apache License 2.0
Stable Diffusion [59]
CreativeML Open RAIL-M
COCO dataset [39]
CC BY 4.0
Localized Narrative dataset [51]
CC BY 4.0
DiffusionDB [77]
MIT License CC BY 4.0
Whoops [6]
CountBench [47] (LAION-400M subset)
CC BY 4.0
GPT3.5 [46]
OpenAl Terms of Use Meta LLaMA3 License
LLaMA3 [1]
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer:
[Yes]
Justification:
The main claims and contributions are clearly stated in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Appendix K.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We don't 't make theoretical contribution.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include implementation details in main paper Sec. 4.3, and further add more details in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We include code in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipp cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtppcccpppccceeccccccccccccccat / ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We include implementation details in main paper Sec. 4.3, and further add more details in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We report single-run results for all the experiments following previous work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in the implementation details in Appendix H.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : htt////errpp...eee http///urss.ccppuuliiieeppc wtt.ehhhhhciieeeeeeeeeeeeeeeeeee ines?
Answer: [Yes]
Justification:
This research conforms every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., g., if there special consider- ation due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix K.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited previous work properly in Appendix L and strictly followed the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We'll release our code, data and model checkpoints upon acceptance under CC license.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper doesn't involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper doesn't involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "UdO8jYbSzs", "reviewer": "Reviewer_dhqF", "summary": "The goal of this paper is to improve the faithfulness of text-to-image generation models. New datasets and fine-tuning frameworks are introduced to address this limitation. Specifically, this paper first adopts LLMs to generate multiple datasets of text prompts that can teach different skills and then generates images with a T2I image based on the text prompts. In the second stage, the LoRA finetuning is performed on the generated datasets to get different experts. Finally, different LoRA experts are merged into one better T2I model. The experiments show better performance on public benchmark datasets.", "questions": "See the weakness.", "limitations": "Limitations are discussed in the appendix!", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The experimental results show consistent improvement over multiple base models. \n\n2. The qualitative results look convincing. The proposed model seems to be more faithful than the baselines.", "weaknesses": "1. The technical novelty is a bit weak. LoRA fine-tuning and merging experts are not original. The prompt generation seems to be straightforward too. \n\n2. Clarity of this paper should be improved. What is the definition of \"Skill\"? Is it referred to different prompt styles? Why do the self-generated images improve the T2I model, especially when the T2I model fails to generate faithful images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Below we address your questions with further clarifications.\n\n> **W1: The technical novelty is a bit weak. LoRA fine-tuning and merging experts are not original. The prompt generation seems to be straightforward too.**\n\nWe would like to first clarify that our main contribution is **introduction of a framework that improves T2I models’ text faithfulness with automatically generated skill-specific data and merge of skill-specific experts** (L40-47). Although LoRA fine-tuning / expert merging / prompt generation methods were initially proposed by previous works, they were not used for improving text faithfulness for T2I models in different skills. Moreover, we would like to elaborate our technical contributions (in relation to previous works):\n- **1. We propose a novel method to automatically create skill-specific image-text pairs based on LLM and the target T2I model itself.** Previous works collect image-text pairs via human annotations (L90-98), and our method significantly reduces the cost for data collection that used to be done via human annotation. Compared to concurrent work relying on heavy image filtering (DreamSync [69]), our data generation method is significantly more efficient by using only 2% of image-text pairs (L98-104).\n- **2. We introduce skill-specific T2I expert learning and merging.** Previous works uses LoRA based T2I model fine-tuning (DreamSync [69]) does not use skill-specific expert learning and merging. ZipLoRA [65] merges two LoRAs for T2I model, but their two LoRAs are limited to a specific subject (e.g., a dog) and a style (e.g., watercolor painting), while we are the first to show the effectiveness of LoRA merging on multiple diverse skills (from 5 datasets) in T2I models (L164-167).\n- **3. We provide comprehensive experiments**, including improvements in two text faithfulness benchmarks (TIFA/DSG), three human preference metrics (Pick-a-Pic/ImageReward/HPS) across three T2I backbones (SD v1.4/v2/XL), human evaluation, ablation studies on design choices, and weak-to-strong generalization (L68-79).\n\n> **W2-1: What is the definition of \"Skill\"? Is it referred to different prompt styles?**\n\nAs we give examples in Fig 1., L121-L122, and L177-178, we use “skills” to refer to different aspects of text prompts that require different writing styles or reasoning capabilities. This includes understanding common objects (e.g., puppy in a backyard), handling long prompts (e.g,. an elegant room with floor-to-ceiling bookshelves, filled with an impressive collection of books of all genres. The cozy reading nook by the window invites anyone to curl up with a good book.”), and displaying commonsense-defying scenes (e.g., cat flying over sky). Following your suggestion, we will add the definition of skill in the introduction section.\n\n> **W2-2: Why do the self-generated images improve the T2I model, especially when the T2I model fails to generate faithful images?**\n\nAs we described in L136-142, we conjecture that the T2I model have seen many prompts that require different skills during pre-training, but the T2I model does not have incentive to demonstrate such skills as it is not important in optimizing loss during the pre-training stage. Our fine-tuning stage aims to efficiently extract the knowledge, which we believe was already inside the T2I models, with automatically generated skill-specific image-text pairs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnGrwOn5kf", "reviewer": "Reviewer_xcmQ", "summary": "The paper analyzes the merging of skill-specific LoRA-finetuned models, trained on generated data and compares this approach to other training approaches (i.e. finetuning, PPO). Moreover, the paper compares the use of GT images and prompts to the use of generated data. The results suggest that this approach performs better than the baseline approaches. Finally, the paper shows a weak-to-strong generalization from weaker models.", "questions": "See weaknesses.", "limitations": "Limitations are provided.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The presented approach shows clear advantages over other methods for alignment - both on text faithfulness and human preference. The paper is easy to read and to understand. Both the effectiveness of auto-generated data (text and images) as well as the merging of experts approach is well studied and ablated.", "weaknesses": "_Weak-to-Strong Generalization_\n\nThe claim in this section doesn't mention the fact that the text here comes from an additional strong model. While the generative model is weaker, the use of an LLM here makes this experiment less convincing. One possible way to improve this section is by ablating here separately again the text and the images.  \n\n_Ablation of LoRA parameters_ \n\nWill a larger $r$ value for the LoRA remove the need for the merging? The paper can benefit from ablation of the size of the bottleneck in the LoRA, together with an experiment that shows that the merging is better than one larger LoRA, trained on all the generated data together. \n\n_Optimization time comparisons_\n\nAn additional concern for the merged LoRAs approach is its optimization time. Can you provide the comparisons for the training time and iteration number for the presented approach and baselines?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Below we address your questions with further clarifications and experiments.\n\n> **W1. Weak-to-Strong Generalization**\n\nRegarding experiments with weaker LM, we would like to bring your attention to Table 9, where we experiment with a LLaMA 3 (8B), which is a publicly available LM known as weaker than GPT-3.5. We find that fine-tuning SDXL with data generated with LLaMA 3 achieves 78.6% on average on DSG, improving the baseline by 5.3% on DSG, closing the gap to GPT-3.5 based results. This demonstrates that SELMA is flexible and compatible even with weaker (but publicly available) prompt generator LMs. Besides, we further experiment with fine-tuning SDXL with data generated with both a weaker image generator SDv2 and a weaker prompt generator LLaMA3. Our results in the table below show that this model achieves similar performance as the model fine-tuned with images generated with SDXL, demonstrating that weak-to-strong generalization holds with weaker data generators.\n\n\n| Base Model | Prompt Generator | Image Generator | DSG  |\n|------------|------------------|-----------------|------|\n| SDXL       | -                | -               | 73.3 |\n| SDXL       | Llama3           | SDv2            | 78.0 |\n| SDXL       | Llama3           | SDXL            | 78.6 |\n| SDXL       | GPT3.5           | SDv2            | 81.3 |\n| SDXL       | GPT3.5           | SDXL            | 80.2 |\n\n\n\n> **W2. Ablation of LoRA parameters**\n\nFollowing your suggestion, we experiment with LoRA with different ranks (128 / 256 / 640) and compare with our default LoRA merging (rank=128).\n\n\n| Approach     | LoRA Rank | DSG  |\n|--------------|-----------|------|\n| SDv2    |           | 70.3 |\n| Single LoRA  | 128       | 74.4 |\n| Single LoRA  | 256       | 74.9 |\n| Single LoRA  | 640       | 71.5 |\n| LoRA Merging (ours) | 128       | **77.7** |\n\n\nThe table shows that increasing the rank of LoRA from 128 to 256 slightly improves the performance (i.e., 74.9 vs. 74.4), but further scaling the rank of the LoRA to 640 significantly drops the performance (i.e., 71.5 vs. 74.4). The performance drop when using LoRA with higher ranks (i.e., rank=640) is similar to the observation in Figure 3 in [A]. This result indicates the effectiveness of our skill-specific learning and merging of LoRA experts. We will add this additional result in the final version.\n\n> **W3. Optimization time comparisons**\n\nIn short, there is no meaningful time difference between training (1) single LoRA for multiple skill-specific image-text pairs vs. (2) learning skill-specific LoRAs for each skill-specific image-text pair and merging them, since we let the T2I model see each example for the same amount of times (i.e., same epochs), in both settings.\n\nFor (1) “single LoRA for multiple skill-specific image-text pairs”, we train the LoRA for 25K training steps on 5K image-text pairs, which takes around 30h on one single L40 GPU.\n\nFor (2) “learning skill-specific LoRAs for each skill-specific image-text pair and merging them” (ours), we train each LoRA for 5K training steps on each of 1K image-text pairs. As shown in Appendix L721-722, this takes around 6h x 5 = 30h on one single L40 GPU (if run parallel in 5 processes, this actually takes 1/5 times of (1)). The LoRA merging takes 26s and is used only once when loading the model before inference, which is negligible compared to the training time.\n\n[A] He et al. (2024), \"Sparse Matrix in Large Language Model Fine-tuning\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2hzYjkdJVe", "reviewer": "Reviewer_c1fe", "summary": "This article goes through four stages: (1) collecting skill-specific prompts using in-context learning of LLMs, (2) self-generating image-text samples for diverse skills without the need for human annotation or feedback from reward models, (3) fine-tuning the expert T2I models on these datasets separately, and (4) obtaining the final model by merging experts from each dataset for efficient adaptation to different skills and mitigation of knowledge conflict in joint training. It is found that the model can optimize itself to an excellent level relying solely on the prompts from the LLM and the model's own generative capabilities.", "questions": "See weaknesses.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This article demonstrates through experiments an interesting conclusion: the model, relying solely on the images generated by prompts, can still be trained on certain domains and achieve superior results without any additional annotation information.\n\n2. This article proves that T2I models struggle with LoRA to accommodate distinct skills and writing styles from different datasets and proposes that using a training-free multi-LoRA fusion method for LoRA trained on different tasks can effectively alleviate this issue.\n\n3. This article demonstrates that even when using data generated by a weaker T2I model, it is still possible to enhance the performance of a stronger model.", "weaknesses": "1. The article does not provide a detailed explanation of the comparison in Table 2 regarding single LoRA. In the comparison between multi-LoRA and single LoRA, are the parameter counts of multi-LoRA and single LoRA the same, or is each LoRA within multi-LoRA equivalent in parameter count to single LoRA? Additionally, is the total training step count for multiple LoRAs in multi-LoRA equal to the training step count for single LoRA, or is the training step count for single LoRA consistent with that of each LoRA within multi-LoRA?\n\n2. The fusion mechanism among multiple LoRAs does not seem to have been thoroughly explored through sufficient ablation experiments. If, after training each LoRA separately in multi-LoRA, a router is introduced to perform gating operations on multi-LoRA, similar to MoE-LoRA, would the effectiveness improve?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Below we address your questions with further clarifications and experiments.\n\n> **W1-1. In the comparison between multi-LoRA and single LoRA, are the parameter counts of multi-LoRA and single LoRA the same, or is each LoRA within multi-LoRA equivalent in parameter count to single LoRA?**\n\nThe single LoRA that learns multiple skills has the same number of parameters for each skill-specific LoRA. Furthermore, we experiment with LoRA with more parameters by using higher ranks (256 / 640) and compare with our default LoRA merging (rank=128).\n\n\n| Approach     | LoRA Rank | DSG  |\n|--------------|-----------|------|\n| SDv2    |           | 70.3 |\n| Single LoRA  | 128       | 74.4 |\n| Single LoRA  | 256       | 74.9 |\n| Single LoRA  | 640       | 71.5 |\n| LoRA Merging (ours) | 128       | **77.7** |\n\nThe table shows that increasing the rank of LoRA from 128 to 256 slightly improves the performance (i.e., 74.9 vs. 74.4), but further scaling the rank of the LoRA to 640 significantly drops the performance (i.e., 71.5 vs. 74.4). The performance drop when using LoRA with higher ranks (i.e., rank=640) is similar to the observation in Figure 3 in [A]. This result indicates the effectiveness of our skill-specific learning and merging of LoRA experts. We will add this additional result in the final version.\n\n> **W1-2. Additionally, is the total training step count for multiple LoRAs in multi-LoRA equal to the training step count for single LoRA, or is the training step count for single LoRA consistent with that of each LoRA within multi-LoRA?**\n\nWe let the T2I model see each example for the same amount of times (i.e., same epochs), in both settings.\n\nFor (1) “single LoRA for multiple skill-specific image-text pairs”, we train the LoRA for 25K training steps on 5K image-text pairs with batch size 64.\n\nFor (2) “learning skill-specific LoRAs for each skill-specific image-text pair and merging them” (ours), we train each LoRA for 5K training steps on each of 1K image-text pairs with batch size 64. \n\n\n> **W2. If, after training each LoRA separately in multi-LoRA, a router is introduced to perform gating operations on multi-LoRA, similar to MoE-LoRA, would the effectiveness improve?**\n\nAs described in L741-744, we freeze the learned LoRA weights and only fine-tune the gating function (i.e., router), which is the setting you mentioned. Additionally, we experiment with learning LoRA weights along with the router. As shown in the below table, learning LoRA weights along with the router achieves worse performance than when LoRAs were frozen. Our default expert merging method – LoRA merging – performs the best. We will add this additional result in the final version.\n\n| Approach         | DSG  |\n|------------------|------|\n| SDv2         | 70.3 |\n| MoE-LoRA (learning router and LoRAs from scratch) | 75.9 |\n| MoE-LoRA (learning router only; LoRAs are frozen)  | 77.2 |\n| LoRA Merging (default)  | **77.7** |\n\n[A] He et al. (2024), \"Sparse Matrix in Large Language Model Fine-tuning\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CTQWvH2Dc0", "reviewer": "Reviewer_WQ94", "summary": "The paper proposes a workflow in which synthetic Text To Image data is generated in order to improve faithfulness of T2I models. Specifically, they generate prompts with LLMs, then Images with T2I models and finally fine-tune pre-trained T2I models with LORA fine-tuning. They fine-tune multiple LORA experts and then merge them by merging parameters. On a limited set of evals, the proposed model achieves good results.", "questions": "Given the complex setup of the proposed workflow, there is a large number of design decisions to make. Do you have guidance, experiments which part of the workflow is the most sensitive with respect to downstream results, e.g., LLM choice, in-context example selection, LLM prompt choice, T2I choice, LORA parameters, etc.?", "limitations": "The limitations are only discussed in the appendix. I think it would be important to have a comprehensive discussion of limitations in the main text.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the faithfulness of Text To Image models. With the increasing application of T2I models across many domains, faithfulness becomes and increasingly important research topic in both industry as well as the academic community. \n\nI appreciate the discussions of related works and how the paper at hand differs from prior art. \n\nWhile the paper motivates its contributions with faithfulness, the proposed workflow is actually independent from faithfulness itself. Faithfulness is only a side effect of the proposed method which has the main focus on multi-task fine-tuning for T2I models. As such, the impact could be even bigger, if the method gets applied to other challenges in the field of T2I.", "weaknesses": "The strength mentioned above is also a weakness of the paper. The paper is motivated by faithfulness. However, the proposed method tackles faithfulness at best as a side effect. While this means that the method might be more general, it also negatively impacts the clarity of the paper and ease of understanding. It might be better to just focus on the multi-task fine-tuning task instead of the impact of individual tasks or side-effects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Below we address your questions with further clarifications and experiments.\n\n> **W1. The strength mentioned above is also a weakness of the paper. The paper is motivated by faithfulness. However, the proposed method tackles faithfulness at best as a side effect. While this means that the method might be more general, it also negatively impacts the clarity of the paper and ease of understanding. It might be better to just focus on the multi-task fine-tuning task instead of the impact of individual tasks or side-effects.**\n\nWe would like to clarify that our framework is not only motivated by text faithfulness but also designed to directly tackle and improve multiple aspects (i.e., skills) of faithfulness. To improve multiple different aspects of faithfulness, our framework (1) automatically generates datasets for teaching faithfulness in different aspects and (2) obtains a T2I model with better text faithfulness, via skill-specific expert learning and expert merging.\n\n> **Q1. Given the complex setup of the proposed workflow, there is a large number of design decisions to make. Do you have guidance, experiments which part of the workflow is the most sensitive with respect to downstream results, e.g., LLM choice, in-context example selection, LLM prompt choice, T2I choice, LORA parameters, etc.?**\n\nThanks for the question. Below we provide ablation results of two design choices: Skill-specific LoRA merging and prompt generator LLM choices. Some of the analysis are from paper Table 2 and Table 9. \n\n(1) First, we show that skill-specific LoRA merging is crucial for mitigating knowledge conflict and improving text faithfulness of T2I models. Merging skill-specific experts during inference outperforms single-LoRA by 3.3% on the DSG benchmark. Besides, merging skill-specific experts also works better than fine-tuning a single LoRA with more parameters (i.e., larger rank), demonstrating the effectiveness of learning separate LoRA to avoid knowledge conflict. \n\n| Approach     | LoRA Rank | DSG  |\n|--------------|-----------|------|\n| SDv2     |           | 70.3 |\n| Single LoRA  | 128       | 74.4 |\n| Single LoRA  | 256       | 74.9 |\n| Single LoRA  | 640       | 71.5 |\n| LoRA Merging (ours) | 128       | **77.7** |\n\n(2) Second, we show the effectiveness of stronger prompt generator LLMs. We collect 5K skill-specific prompts with Llama-3 and GPT-3.5, and fine-tune SDXL with self-generated images. The table below shows that learning from prompts generated with GPT-3.5 outperforms learning from prompts generated with Llama-3 (80.2 vs. 78.6 on DSG benchmark), demonstrating a stronger LLM can help improve the performance. However, note that both SDXL models fine-tuned with GPT-3.5 and Llama-3 prompts outperform the original SDXL baseline by a large margin, demonstrating the effectiveness of our pipeline even with weaker LLMs. \n\n| Approach | Prompt Generator | DSG  |\n|----------|------------------|------|\n| SDXL | -                | 73.3 |\n| SDXL     | Llama-3          | 78.6 |\n| SDXL     | GPT-3.5          | **80.2** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their valuable feedback. We also appreciate that they acknowledge SELMA's strengths:
- Clear advantages over other methods for text-image alignment (Reviewer dhqF, xcmQ, WQ94)
- Our proposed automatic data generation pipeline and LoRA merging approach are well studied and ablated (Reviewer xcmQ, c1fe)
- Interesting findings supported with experiments (e.g., weak-to-strong generalization, learning from self-generated images) (Reviewer c1fe).

We have addressed all the questions in our rebuttal, and will incorporate the feedback in the final version.

Meta-review: AC, along with the reviewers, holds a positive opinion towards SELMA. Although the novelty level is being questioned at first, the authors made good attempt to argue this way of using LLM's capability to conduct skill specific text 2 image data augmentation is unique and novel. While presentation could be improved, as suggested by multiple reviewers, AC thinks the current way of presentation, as it is simply a very engineering idea, is already decent. AC strongly suggests the authors take the review comments, especially WQ94's weakness point, into the final draft's revision. Overall, I would recommend Accept (poster).
Decision: Accept (poster)