Title: SELMA: Learning and Merging Skill-Specific Text-to-Image Experts with Auto-Generated Data
Venue: NeurIPS 2024
Accepted: yes
Keywords: Text-to-Image Generation; LoRA Learning and Merging; Skill-based Experts; Auto-Generated Data
Abstract: Recent text-to-image (T2I) generation models have demonstrated impressive capabilities in creating images from text descriptions. However, these T2I generation models often fail to generate images that precisely match the details of the text inputs, such as incorrect spatial relationship or missing objects. In this paper, we introduce SELMA: Skill-Specific Expert Learning and Merging with Auto-Generated Data, a novel paradigm to improve the faithfulness of T2I models by fine-tuning models on automatically generated, multi-skill image-text datasets, with skill-specific expert learning and merging. First, SELMA leverages an LLM’s in-context learning capability to generate multiple datasets of text prompts that can teach different skills, and then generates the images with a T2I model based on the prompts. Next, SELMA adapts the T2I model to the new skills by learning multiple single-skill LoRA (low-rank adaptation) experts followed by expert merging. Our independent expert fine-tuning specializes multiple models for different skills, and expert merging helps build a joint multi-skill T2I model that can generate faithful images given diverse text prompts, while mitigating the knowledge conflict from different datasets. We empirically demonstrate that SELMA significantly improves the semantic alignment and text faithfulness of state-of-the-art T2I diffusion models on multiple benchmarks (+2.1% on TIFA and +6.9% on DSG), human preference metrics (PickScore, ImageReward, and HPS), as well as human evaluation. Moreover, fine-tuning with image-text pairs auto-collected via SELMA shows comparable performance to fine-tuning with ground truth data. Lastly, we show that fine-tuning with images from a weaker T2I model can help improve the generation quality of a stronger T2I model, suggesting promising weak-to-strong generalization in T2I models. We provide code in the supplementary materials.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a workflow in which synthetic Text To Image data is generated in order to improve faithfulness of T2I models. Specifically, they generate prompts with LLMs, then Images with T2I models and finally fine-tune pre-trained T2I models with LORA fine-tuning. They fine-tune multiple LORA experts and then merge them by merging parameters. On a limited set of evals, the proposed model achieves good results.\n\nSTRENGTHS:\nThe paper addresses the faithfulness of Text To Image models. With the increasing application of T2I models across many domains, faithfulness becomes and increasingly important research topic in both industry as well as the academic community. \n\nI appreciate the discussions of related works and how the paper at hand differs from prior art. \n\nWhile the paper motivates its contributions with faithfulness, the proposed workflow is actually independent from faithfulness itself. Faithfulness is only a side effect of the proposed method which has the main focus on multi-task fine-tuning for T2I models. As such, the impact could be even bigger, if the method gets applied to other challenges in the field of T2I.\n\nWEAKNESSES:\nThe strength mentioned above is also a weakness of the paper. The paper is motivated by faithfulness. However, the proposed method tackles faithfulness at best as a side effect. While this means that the method might be more general, it also negatively impacts the clarity of the paper and ease of understanding. It might be better to just focus on the multi-task fine-tuning task instead of the impact of individual tasks or side-effects.\n\nQUESTIONS:\nGiven the complex setup of the proposed workflow, there is a large number of design decisions to make. Do you have guidance, experiments which part of the workflow is the most sensitive with respect to downstream results, e.g., LLM choice, in-context example selection, LLM prompt choice, T2I choice, LORA parameters, etc.?\n\nLIMITATIONS:\nThe limitations are only discussed in the appendix. I think it would be important to have a comprehensive discussion of limitations in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article goes through four stages: (1) collecting skill-specific prompts using in-context learning of LLMs, (2) self-generating image-text samples for diverse skills without the need for human annotation or feedback from reward models, (3) fine-tuning the expert T2I models on these datasets separately, and (4) obtaining the final model by merging experts from each dataset for efficient adaptation to different skills and mitigation of knowledge conflict in joint training. It is found that the model can optimize itself to an excellent level relying solely on the prompts from the LLM and the model's own generative capabilities.\n\nSTRENGTHS:\n1. This article demonstrates through experiments an interesting conclusion: the model, relying solely on the images generated by prompts, can still be trained on certain domains and achieve superior results without any additional annotation information.\n\n2. This article proves that T2I models struggle with LoRA to accommodate distinct skills and writing styles from different datasets and proposes that using a training-free multi-LoRA fusion method for LoRA trained on different tasks can effectively alleviate this issue.\n\n3. This article demonstrates that even when using data generated by a weaker T2I model, it is still possible to enhance the performance of a stronger model.\n\nWEAKNESSES:\n1. The article does not provide a detailed explanation of the comparison in Table 2 regarding single LoRA. In the comparison between multi-LoRA and single LoRA, are the parameter counts of multi-LoRA and single LoRA the same, or is each LoRA within multi-LoRA equivalent in parameter count to single LoRA? Additionally, is the total training step count for multiple LoRAs in multi-LoRA equal to the training step count for single LoRA, or is the training step count for single LoRA consistent with that of each LoRA within multi-LoRA?\n\n2. The fusion mechanism among multiple LoRAs does not seem to have been thoroughly explored through sufficient ablation experiments. If, after training each LoRA separately in multi-LoRA, a router is introduced to perform gating operations on multi-LoRA, similar to MoE-LoRA, would the effectiveness improve?\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
The paper analyzes the merging of skill-specific LoRA-finetuned models, trained on generated data and compares this approach to other training approaches (i.e. finetuning, PPO). Moreover, the paper compares the use of GT images and prompts to the use of generated data. The results suggest that this approach performs better than the baseline approaches. Finally, the paper shows a weak-to-strong generalization from weaker models.

STRENGTHS:
The presented approach shows clear advantages over other methods for alignment - both on text faithfulness and human preference. The paper is easy to read and to understand. Both the effectiveness of auto-generated data (text and images) as well as the merging of experts approach is well studied and ablated.

WEAKNESSES:
_Weak-to-Strong Generalization_

The claim in this section doesn't mention the fact that the text here comes from an additional strong model. While the generative model is weaker, the use of an LLM here makes this experiment less convincing. One possible way to improve this section is by ablating here separately again the text and the images.  

_Ablation of LoRA parameters_ 

Will a larger $r$ value for the LoRA remove the need for the merging? The paper can benefit from ablation of the size of the bottleneck in the LoRA, together with an experiment that shows that the merging is better than one larger LoRA, trained on all the generated data together. 

_Optimization time comparisons_

An additional concern for the merged LoRAs approach is its optimization time. Can you provide the comparisons for the training time and iteration number for the presented approach and baselines?

QUESTIONS:
See weaknesses.

LIMITATIONS:
Limitations are provided.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

comments: SUMMARY:
The goal of this paper is to improve the faithfulness of text-to-image generation models. New datasets and fine-tuning frameworks are introduced to address this limitation. Specifically, this paper first adopts LLMs to generate multiple datasets of text prompts that can teach different skills and then generates images with a T2I image based on the text prompts. In the second stage, the LoRA finetuning is performed on the generated datasets to get different experts. Finally, different LoRA experts are merged into one better T2I model. The experiments show better performance on public benchmark datasets.

STRENGTHS:
1. The experimental results show consistent improvement over multiple base models. 

2. The qualitative results look convincing. The proposed model seems to be more faithful than the baselines.

WEAKNESSES:
1. The technical novelty is a bit weak. LoRA fine-tuning and merging experts are not original. The prompt generation seems to be straightforward too. 

2. Clarity of this paper should be improved. What is the definition of "Skill"? Is it referred to different prompt styles? Why do the self-generated images improve the T2I model, especially when the T2I model fails to generate faithful images?

QUESTIONS:
See the weakness.

LIMITATIONS:
Limitations are discussed in the appendix!
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16