Stylus: Automatic Adapter Selection for Diffusion Models

Authors: michael  luo
justin  wong
brandon  trabucco
yanping  huang
joseph e gonzalez
zhifeng  chen
ruslan  salakhutdinov
ion  stoica
u c berkeley
cmu  mld
google  deepmind
lora  textual
inversion  hypernetwork
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:35:37

Abstract: Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters—most of which are highly customized with insufficient descriptions. To generate high quality images, this paper explores the problem of matching the prompt to a Stylus of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP/FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model.

Main paper: 
 introduction: 
 In the evolving field of generative image models, finetuned adapters [7, 11] have become the standard, enabling custom image creation with reduced storage requirements. This shift has spurred the growth of extensive open-source platforms that encourage communities to develop and share different adapters and model checkpoints, fueling the proliferation of creative AI art [28, 51] . As the ecosystem expands, the number of adapters has grown to over 100K, with Low-Rank Adaptation (LoRA) [14] emerging as the dominant finetuning approach (see Fig. 3 ). A new paradigm has emerged where users manually select and creatively compose multiple adapters, on top of existing checkpoints, to generate high-fidelity images, moving beyond the standard approach of improving model class or scale.In light of performance gains, our paper explores the automatic selection of adapters based on user-provided prompts (see Fig. 1 ). However, selecting relevant adapters presents unique challenges compared to existing retrieval-based systems, which rank relevant texts via lookup embeddings [18] . Specifically, efficiently retrieving adapters requires converting adapters into lookup embeddings, a step made difficult with low-quality documentation or no direct access to training data-a common issue on open-source platforms. Furthermore, in the context of image generation, user prompts often imply multiple highly-specific tasks. For instance, the prompt "two dogs playing the snow" suggests that there are two tasks: generating images of "dogs" and "snow". This necessitates segmenting the prompt into various tasks (i.e. keywords) and selecting relevant adapters for each task, a requirement beyond the scope of existing retrieval-based systems [9] . Finally, composing multiple adapters can degrade image quality, override existing concepts, and introduce unwanted biases into the model (see App. A.4).We propose Stylus, a system that efficiently assesses user prompts to retrieve and compose sets of highly-relevant adapters, automatically augmenting generative models to produce diverse sets of high quality images. Stylus employs a three-stage framework to address the above challenges. As shown in Fig. 2 , the refiner plugs in an adapter's model card, including generated images and prompts, through a multi-modal vision-language model (VLM) and a text encoder to pre-compute concise adapter descriptions as lookup embeddings. Similar to prior retrieval methods [18] , the retriever scores the relevance of each embedding against the user's entire prompt to retrieve a set of candidate adapters. Finally, the composer segments the prompt into disjoint tasks, further prunes irrelevant candidate adapters, and assigns the remaining adapters to each task. We show that the composer identifies highly-relevant adapters and avoids conceptually-similar adapters that introduce biases detrimental to image generation ( § 4.3). Finally, Stylus applies a binary mask to control the number of adapters per task, ensuring high image diversity by using different adapters for each image and mitigating challenges with composing many adapters. To evaluate our system, we introduce StylusDocs, an adapter dataset consisting of 75K LoRAs1 , that contains pre-computed adapter documentations and embeddings from Stylus's refiner. Our results demonstrate that Stylus improves visual fidelity, textual alignment, and image diversity over popular Stable Diffusion (SD 1.5) checkpoints-shifting the CLIP-FID Pareto curve towards greater efficiency and achieving up to 2x higher preference scores with humans and vision-language models (VLMs) as evaluators. As a system, Stylus is practical and does not present large overheads to the batch image generation process. Finally, Stylus can extend to different image-to-image application domains, such as image inpainting and translation. ing multiple Low-Rank adapters (LoRA), the popular approach within existing open-source communities [28, 29, 51] .Adapter composition has emerged as a crucial mechanism for enhancing the capabilities of foundational models across various applications [19, 36, 40, 45, 46] . For large language models (LLM), the linear combination of multiple adapters improves in-domain performance and cross-task generalization [3, 15, 16, 48, 49, 55] . In the image domain, merging LoRAs effectively composes different tasks-concepts, characters, poses, actions, and styles-together, yielding images of high fidelity that closely align with user specifications [25, 56] . Adapters also play a key role in synthetic data methods in few-shot computer vision [47] . Our approach advances this further by actively segmenting user prompts into distinct tasks and merging the appropriate adapters for each task.Retrieval-based Methods. Retrieval-based methods, such as retrieval-augmented generation (RAG), significantly improve model responses by adding semantically similar texts from a vast external database [18] . These methods convert text to vector embeddings using text encoders, which are then ranked against a user prompt based on similarity metrics [4, 9, 21, 27, 37, 39] . Similarly, our work draws inspiration from RAG to encode adapters as vector embedings: leveraging visual-language foundational models (VLM) to generate semantic descriptions of adapters, which are then translated into embeddings. A core limitation to RAG is limited precision, retrieving semi-relevant documents that do not exactly answer the prompt. This leads to a "needle-in-the-haystack" problem, where more relevant documents are buried further down the list [9] . Recent work introduce reranking step; this technique uses cross-encoders to assess both the raw user prompt and the ranked set of raw texts individually, thereby discovering texts based on actual relevance [27, 38] . Rerankers have been successfully integrated with various LLM-application frameworks [2, 24, 35] . 
our method: stylus: 
 Adapter selection presents three distinct challenges compared to existing methods for retrieving text documents, as outlined in Section 2. First, computing embeddings for adapters is a novel task, made more difficult without access to training datasets. Furthermore, in the context of image generation, user prompts often specify multiple highly fine-grained tasks. This challenge extends beyond retrieving relevant adapters relative to the entire user prompt, but also matching them with specific tasks within the prompt. Finally, composing multiple adapters can degrade image quality and inject foreign biases into the model. Our three-stage framework below-Refine, Retrieve, and Compose-addresses the above challenges (Fig. 2 ).The refiner is a two-stage pipeline designed to generate textual descriptions of an adapter's task and the corresponding text embeddings for retrieval purposes. This approach is analagous to pre-computed embeddings over an external database of texts in retrieval-based methods [18] . Given an adapter A i , the first stage is a vision-language model (VLM) that takes in the adapter's model card-a set of randomly sampled example images from the model card I i ∈ {I i1 , I i2 , ...}, the corresponding prompts P i ∈ {p i1 , p i2 , ...}, and an author-provided description,2 D i -and returns an improved description D * i . Optionally, the VLM also recommends the weight for LoRA-based adapters, Stylus produces highly detailed images that correctly depicts keywords in the context of the prompt. For the prompt "A graffiti of a corgi on the wall", our method correctly depicts a spray-painted corgi, whereas the checkpoint generates a realistic dog.as the adapter weight is usually specified either in the author's description D i or the set of prompts P i , a feature present in popular image generation software [1] . We denote this weight/coefficient as α i . If information cannot be found, the LoRA's weight is set to α i = 0.8. In our experiments, these improved descriptions were generated by Gemini Ultra [43] (see § A.1 for prompt). We chose the Gemini class of models since it has mature safety guardrailing. Specifically, Google's VertexAI API provides stringent safety settings to block explicit content for the input prompt. Safety filters helped us filter out around 30% of original adapters that were tagged as non-explicit by other model repositories.The second stage uses an embedding model (E) to embed the text description D * i for each adapters to yield embeddings, e i = E(D * ) . In our experiments, we create embeddings from OpenAI's text-embedding-3-large model [21, 30] . We store pre-computed embeddings in a vector database, formally notated by the matrix, V .The retriever fetches the most relevant adapters over the entirety of the user's prompt using cosine similarity. Precisely, the retriever employs the same embedding model (E) to process the user prompt, q, generating embedding e q = E(q). Using the vector database, we calculate exact cosine similarity scores between the prompt's embedding e s and the embedding of each adapter in the matrix V .The similarity vector, s q = e T q V|eq||V | , scores the adapter descriptions by similarity. The retriever simply returns indices of the top-k adapters A k = top-k(s q ). In our experiments, we find k = 150 is effective for StylusDocs. We denote the set of k descriptions of the adapters, A k as D * k .The composer serves a dual purpose: segmenting the prompt into tasks from a prompt's keywords and assigning retrieved adapters to tasks. This implicitly filters out adapters that are not semantically aligned with the prompt and detects those likely to introduce foreign bias to the prompt through keyword grounding. For example, if the prompt is "pandas eating bamboo", the composer may discard an irrelevant "grizzly bears" adapter and a biased "panda mascots" adapter. The composer (C) is a function of the prompt (q), the top K adapters (A K ) from the retriever. Formally, denote the tasks identified by the composer as T (q) = {t 1 , t 2 , . . . , t n }. The composer produces a mapping from task to adapters:C(s, A K ) = {(t i , A ki ) | t i ∈ T (q), A ki ⊆ A K , ∀j ∈ A ki , Align(A j , t i )}(1) where A ki is the subset of adapters per task t i , Align(A j , t i ) is a predicate that holds if the adapter, A j is aligned with the task, t i . While the composer can be further improved by fine-tuning with human-labeled data [34] , we find that prompting a long-context Large Language Model (LLM) suffices. The LLM accepts the adapter descriptions and the prompt as part of its context and returns a mapping of tasks to a curated set of adapters. In practice, the alignment function is determined in the LLM's chain-of-thought procedure before it outputs the final mapping of adapters to tasks. In our implementation, we choose Gemini 1.5, with a 128K context window, as the composer's LLM (see App. A.3 for the full prompt). Stylus's composer is similar to reranking. Rerankers employ cross encoders (F) that compare the retriever's individual adapter descriptions, generated from the refiner, against the user prompt to determine better similarity scores: F(p, D * ). This prunes for adapters based on semantic relevance, thereby improving search quality, but not over keyword alignment. Our experimental ablations ( § 4.3) show that our composer outperforms existing rerankers (Cohere, rerank-english-v2.0) [38] . The composer maps tasks to corresponding sets of highly relevant adapters. To further mitigate sensitivity to low-quality adapters, Stylus reduces the number of selected adapters with a straightforward masking scheme. Specifically, for each task, candidate masks are generated, and one is randomly selected to be applied over the set of adapters. Formally, for a given task, m i ∈ {0, 1} |A k | , is either a one hot encoding, ⃗ 1, or ⃗ 0, forming a set of possible masks, M i . Across all tasks, masks are combined by taking the cross-product, G = M 1 ×M 2 ×...×M n . The combinatorial sets of masking schemes enable diverse linear combinations of adapters for a single prompt, leading to highly-diverse images ( § 4.2.3). This approach also curtails the number of final adapters merged into the base model, minimizing the risk of composing low-quality adapters that may introduce undesirable effects to the image [56] .Stylus employs two key insights for effectively merging adapter weights. First, when applied to a single task, large adapter weights can introduce notable visual artifacts, such as over-saturation (Fig. 14a ). Second, across multiple tasks, adapters tend to be orthogonal in the weight space, as they are designed to modify distinct, orthogonal concepts [8] . Hence, Stylus computes the final adapter weights by averaging weights per task and summing weights across tasks. This approach ensures that the adapter weights per task remain appropriately scaled. We mathematically illustrate our merging scheme below. Recall, the refiner outputs α i , the recommended weight/coefficient, for each adapter. ( § 3.1). As shown in recent work [56] , multiple LoRAs can be merged with the base model weights (W base ). We arrive at our final merged model weights by a summing the adapter weights normalized by task. For a mapping, {(t 1 , A k1 ), (t 2 , A k2 ), . . . (t n , A kn )}, and g = (m 1 , m 2 , . . . , m n ) ∈ G, the final model weight is:EQUATIONwhere x i = M ask(m i , A i ) and ∆ j is the LoRA's weight. We set β = 0.8 to mitigate image saturation, where assigning high adapter weights to an individual task (or concept) leads to sharp decreases in image quality (see App. A.4). For batch inference, Stylus returns images sorted by CLIP score.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Stylus is not a theory paper and hence does not make theoretical claims. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 Adapter Testbed. Adapter selection requires a large database of adapters to properly evaluate its performance. However, existing methods [15, 55] only evaluate against 50-350 adapters for language-based tasks, which is insufficient for our use case, since image generation relies on highly fine grained tasks that span across many concepts, poses, styles, and characters. To bridge this gap, we introduce StylusDocs, a comprehensive dataset that pulls 75K LoRAs from popular model repositories, Civit AI and HuggingFace [28, 51] . This dataset contains precomputed OpenAI embeddings [21] and improved adapter descriptions from Gemini Ultra-Vision [43] , the output of Stylus's refiner component ( § 3.1). We further characterize the distribution of adapters in App. A.3. Generation Details. We assess Stylus against Stable-Diffusion-v1.5 [40] as the baseline model.Across experiments, we employ two well-known checkpoints: Realistic-Vision-v6, which excels in producing realistic images, and Counterfeit-v3, which generates cartoon and anime-style images. Our image generation process integrates directly with Stable-Diffusion WebUI [1] and defaults to 35 denoising steps using the default DPM Solver++ scheduler [26] . To replicate high-quality images from existing users, we enable high-resolution upscaling to generate 1024x1024 from 512x512 images, with the default latent upscaler [17] and denoising strength set to 0.7. For images generated by Stylus, we discovered adapters could shift the image style away from the checkpoint's original style. To counteract this, we introduce a debias prompt injected at the end of a user prompt to steer images back to the checkpoint's style 3 . We launched 16 replicas of Stylus and Stable Diffusion on 8 A100-80GB GPUs for 4 weeks to generate images for evaluation.To demonstrate our method's general applicability, we evaluate Stylus over a cross product of two datasets, Microsoft COCO [22] and PartiPrompts [53] , and two checkpoints, which generate realistic and anime-style images respectively. Examples of images generated in these styles are displayed in Figure 4 ; Stylus generates highly detailed images that better focus on specific elements in the prompt.To conduct human evaluation, we enlisted four users to assess 150 images from both Stylus and Stable Diffusion v1.5 for each dataset-checkpoint combination. These raters were asked to indicate their preference for Stylus or Stable-Diffusion-v1.5. In Fig. 5 , users generally showed a preference for Stylus over existing model checkpoints. Although preference rates were consistent across datasets, they varied significantly between different checkpoints. Adapters generally improve details to their corresponding tasks (e.g. generate detailed elephants); however, for anime-style checkpoints, detail is less important, lowering preference scores.We assess Stylus using two automatic benchmarks: CLIP [12] , which measures the correlation between a generated images' caption and users' prompts, and FID [13] , which evaluates the diversity and aesthetic quality of image sets. We evaluate COCO 2014 validation dataset, with 10K sampled prompts, and the Realistic-Vision-v6 checkpoint. Fig. 6a shows that Stylus shifts the Pareto curve towards greater efficiency, achieving better visual fidelity and textual alignment. This improvement aligns with our human evaluations, which suggest a correlation between human preferences and the FID scores.We use VLM as a Judge to assess two key metrics: textual alignment and visual fidelity, simulating subjective assessments [5] . For visual fidelity, the VLM scores based on disfigured limbs and unrealistic composition of objects. When asked to make subjective judgements, autoregressive models tend to exhibit bias towards the first option presented. To combat this, we evaluate Stylus under both orderings and only consider judgements that are consistent across reorderings; otherwise, we label it a tie. In Fig. 8a , we assess evaluate 100 randomly sampled prompts from the PartiPrompts dataset [53] . Barring ties, we find visual fidelity achieves 60% win rate between Stylus and the Stable Diffusion realistic checkpoint, which is conclusively consistent with the 68% win rate from our human evaluation. For textual alignment, we find negligible differences between Stylus and the Stable Diffusion checkpoint. As most prompts lead to a tie, this indicates Stylus does not introduce additional artifacts. We provide the full prompt in Appendix A.5.Given identical prompts, Stylus generates highly diverse images due to different composer outputs and masking schemes. Qualitatively, Fig. 7 shows that Stylus generates dragons, maidens, and kitchens in diverse positions, concepts, and styles. To quantitatively assess this diversity, we use two metrics: dFID: Previous evaluations with FID [13] show that Stylus improves image quality and diversity across prompts 4 . We define dFID specifically to evaluate diversity per prompt, calculated as the variance of latent embeddings from InceptionV3 [42] . Mathematically, dFID involves fitting a Normal distribution N (µ, Σ) to the latent features of InceptionV3, with the metric given by the trace of the covariance matrix, dFID = Tr Σ. GPT-4V: We use VLM as a Judge to assess image diversity between images generated using Stylus and the Stable Diffusion checkpoint over PartiPrompts. Five images are sampled per group, Stylus and SD v1.5, with group positions randomly swapped across runs to avoid GPT-4V's positional bias [56] .Similar to VisDiff, we ask GPT-4V to rate on a scale from 0-2, where 0 indicates no diversity and 2 indicates high diversity [6] . Full prompt and additional details are provided in App A.5. Fig. 8b displays preference rates and defines a win when Stylus achieves higher dFID or receives a higher score from GPT-4V for a given prompt. Across 200 prompts, Stylus prevails in approximately 60% and 58% cases for dFID and GPT-4V respectively, excluding ties. Figure 8c Gemini-Ultra Refiner: This refiner, used throughout all our experiments, employs Gemini-Ultra to auto-generate enhanced adapter descriptions, improving both relevance and specificity. Consequently, Stylus attains better CLIP and FID scores than SDv1.5. GPT-4o Refiner: The GPT-4o refiner, OpenAI's most advanced model, outputs the best adapter descriptions, yielding the highest performance gains across CLIP and FID scores. This baseline demonstrates that Stylus's end-to-end performance is highly dependent on the quality and specificity of adapter descriptions.We benchmark Stylus's performance relative to different retrieval methods. For all baselines below, we select the top three adapters and merge them into the base model. Random: Adapters are randomly sampled without replacement from StylusDocs. Retriever: The retriever emulates standard RAG pipelines [18, 55] , functionally equivalent to Stylus without the composer stage. Top adapters are fetched via cosine similarity over adapter embeddings. Reranker: An alternative to Stylus's composer, the reranker fetches the retriever's adapters and plugs a cross-encoder that outputs the semantic similarity between adapters' descriptions and the prompt. We evaluate with Cohere's reranker endpoint [38] . As shown in Tab. 6b, Stylus achieves the highest CLIP and FID scores, outperforming all other baselines which fall behind the base Stable Diffusion model. First, both the retriever and reranker significantly underperform compared to Stable Diffusion. Each method selects adapters that are similar to the prompt but potentially introduce unrelated biases. In Fig. 9 , both methods choose adapters related to elephant movie characters, which biases the concept of elephants and results in depictions of unrealistic elephants. Furthermore, both methods incorrectly assign weights to adapters, causing adapters' tasks to overshadow other tasks within the same prompt. In Fig. 9 , both the reranker and retriever generate images solely focused on singular items-beds, chairs, suitcases, or trains-while ignoring other elements specified in the prompt. We provide an analysis of failure modes in A.4.Conversely, the random policy exhibits performance comparable, but slightly worse, to Stable Diffusion. The random baseline chooses adapters that are orthogonal to the user prompt. Thus, these adapters alter unrelated concepts, which does not affect image generation. In fact, we observed that the distribution of random policy's images in Fig. 9 were nearly identical to Stable Diffusion.This section breaks down the latency introduced by various components of Stylus. We note that image generation time is independent of Stylus, as adapter weights are merged into the base model [14] . Figure 11 demonstrates the additional time Stylus contributes to the image generation process across different batch sizes (BS), averaged over 100 randomly selected prompts. Specifically, Stylus adds 12.1 seconds to the image generation time, with the composer accounting for 8.5 seconds. The composer's large overhead is due to long-context prompts, which include adapter descriptions for the top 150 adapters and can reach up to 20K+ tokens. Finally, when the BS is 1, Stylus presents a 75% increase in overhead to the image generation process. However, Stylus's latency remains consistent across all batch sizes, as the composer and retriever run only once. Hence, for batch inference workloads, Stylus incurs smaller overheads as batch size increases.Beyond text-to-image, Stylus applies across various image-to-image tasks. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Yes, our paper discloses the necessary information to reproduce the main results in our paper. We provide the full sample prompts to the Refiner VLM and Composer LLM in the Appendix. We discuss which models (Gemini Ultra-Vision & 1.5) were used to generate the adapter descriptions for StylusDocs and the composer's output adapters. We also concisely describe our experiments, hyperparameters, and datasets, providing the full sample evaluation prompts for VLM as a judge in Tab. 4. Finally, Stylus is open-sourced. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. Justification: In Figure 8 , we show the one standard deviation as a shaded region. We follow existing conventions for reporting the CLIP/FID pareto curves. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We only access public domain adapters and use Google's default safety filter for Gemini models to remove potentially harmful adapters. We also remove all adapters that were tagged as explicit from Civit AI. The quality of our StylusDocs is carefully curated. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: In the conclusion, we note that StylusDocs is carefully curated to avoid explicit and low-quality adapters, but this may not be guaranteed. Further, the work suffers from the same weakness other public domain image generation tools have with improper use and misinformation, including unintentionally producing explicit content and reproducing copyrighted material from the training data. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 The strategic composition and routing of adapters in Stylus introduce a new dimension of model performance, broadening the scope of potential applications. One such application is the automatic creation of agentic workflows [54, 57] . For instance, Stylus's composer can decompose a complex task into a graph of subtasks and assign them to specialized agents to improve end-to-end performance. Additionally, routing can extend beyond adapters to encompass different models, allowing Stylus to optimize the cost-performance tradeoff by dynamically selecting between high-performing, resource-intensive models and more efficient, lower-cost models [31, 33] . Finally,for fact verification, adapters have shown significant potential in reducing hallucinations [10, 44] . Stylus can selectively use domain-specific, fine-tuned models to enhance factual accuracy and better verify claims. As demonstrated in Sec. 4, Stylus demonstrates significant potential for improvement, as adapter composition introduces future research challenges beyond the scope of this work. A summary of Stylus's failure cases are provided in Fig. 14 . Specifically, adapters can restrict certain concepts from appearing in an image and limit diversity among multiple subjects within a scene. While Stylus does not fundamentally solve these challenges, Stylus reduces the likelihood of these problems occurring by reducing the number of adapters through its masking algorithm. Lastly, Stylus introduces noticeable overheads to the inference pipeline, primarily stemming from the composer's long context prompts, which can be accelerated with various sequence parallel techniques [20, 23] . 
conclusion: 
 We propose Stylus, a flexible algorithm that automatically selects and composes adapters to generate better images. Our method leverages a three-stage framework that precomputes adapters as lookup embeddings and retrieves most relevant adapters based on prompts' keywords.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: In App.A.7, we discuss the continual curation process by which we ask community members to report problematic adapters not caught by our initial curation process. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Figure 1 : Adapter Selection. Given a user-provided prompt, our method identifies highly relevant adapters (e.g. Low-Rank Adaptation, LoRA) that are closely aligned with the prompt's context and at least one of the prompt's keywords. Composing relevant adapters into Stable Diffusion improves visual fidelity, image diversity, and textual alignment. Note that these prompts are sampled from MS-COCO [22] . 
 
 Appendix: A.1 Details of the Refiner VLM We provide a complete example input to the refiner's VLM in Tab. 1. The prompt utilizes Chain-of-Thought (CoT) prompting, which decomposes the VLM's goal of producing better adapter descriptions into two steps [50, 52] . Initially, the VLM categorizes the adapter's task into one of several topics-such as concepts, styles, characters, or poses. Subsequently, the VLM is prompted to elaborate on why the adapter is associated with a particular topic and how it modifies images within that context. We found that this two step logical process significantly improved the structure and quality of model responses.A.2 Details of the Composer LLM We provide a full example prompt of the composer's LLM component in Tab. 2, which is plugged through the Gemini 1.5 endpoint [43] . Our experiments feed in descriptions of the top 150 adapters into the LLM's context. Using a Chain-of-Thought (CoT) approach, the prompt is structured to first identify keywords or tasks, then allocate appropriate adapters to these tasks. If necessary, it merges keywords for adapters that span multiple tasks [50, 52] .This section describes StylusDocs, which comprises of 76K Low Rank Adapters (LoRAs) from public repositories, including Civit AI and Hugging Face [28, 51] . We excluded NSFW-labeled adapters from the Civit AI dataset, which originally contained over 100K LoRAs. Figure 13 illustrates the distribution of adapters across various semantic categories and their popularity, measured by download counts. A significant majority, 70%, of adapters belong to the character and celebrity category, primarily consisting of anime or game characters. Another 13% of adapters modify image style, 8% adjust clothing, and 4% represent various concepts (Fig. 13a ). These statistics indicate that our experiments consider a minor proportion of adapters, as the COCO dataset does not feature characters or celebrities [22] . Despite this, Stylus outperforms base Stable Diffusion. Furthermore, the popularity of adapters follows a Pareto distribution, where the top adapters receive exponentially more downloads than the others (Fig. 13a ). However, the top adapter accounts for only 0.5% of total downloads, which suggests that the distribution is long-tailed.We detail different failure modes that were discovered while developing Stylus.Image saturation. The quality of image generation is highly depend on adapters' weights. If the assigned weight is above the recommended value, the adapter negatively impacts image generation, leading to a growing number of visual inconsistencies and artifacts. In Fig. 14a , assigning a high weight to a "James Bond" LoRA increases images exposure and introducing significant visual tearing. Stylus mitigates over-saturation with its refiner component, which extract the right adapter weights from the adapter's model card. Lastly, Stylus uniformly weights adapters based on their associated tasks, ensuring that similar adapters do not significantly impact their corresponding tasks.Task Blocking. Composing adapters presents the risk of overwriting existing concepts or tasks specified in the prompt and other selected adapters. We illustrate several examples in Figure 2 Your goal is improve the description of a model adapter's task for Stable Diffusion, with images, prompts, and descriptions pulled from popular model repositories. Above, we have provided the following information and the associated constraints:1. Examples of generated images (from left to right) from the adapter and the corresponding user-provided prompts.• Some prompts may specify the adapter weight (i.e. <lora:NAME:WEIGHT>). If provided, you will need to infer the adapter's name and weight. Prioritize this weight over the author's recommended weight. 2. The adapter's model card from the original author. This includes the title, tags, trigger words, and description.• The model card description may be incorrect, misleading, or incomplete.• The model card may specify the weight of the model adapter, or the recommended range. Find the recommended weight of the adapter (default is 0.8). % Chain-of-Thought Prompting Again, your mission is to provide a clear description of the model's adapter purpose and its impact on the image. To do so, you should implicitly categorize the model adapter into only one of the following topics: [Concept, Style, Pose, Action, Celebrity/Character, Clothing, Background, Building, Vehicle, Animal, Action]. Do not associate an adapter with a topic that is vague or uninteresting.First, describe the topic associated with the adapter and explain how this adapter alters the images, based on the common elements observed in the example images. Your requirements are: • Do not describe any training or dataset-related details.• Provide additional context from your prior knowledge if there is insufficient information.• Do not hallucinate and repeat text. Output only english words and sentences.Second, recommend an optimal weight for the adapter as a float. Do not specify a range, only give one value.Example 1: [Description of adapter and its weight] train LoRA overrides the toy train concept (left), a park bench LoRA masks a person in an orange blanket (middle), and a fancy cake LoRA erases the image of a man eating the cake (right). Task blocking typically arises from two main issues: the adapter weight set too high or too many adapters merged into the base model. Stylus addresses this by reducing an adapter's weight with uniform weighting per task, while the masking scheme reduces the number of selected adapters. Although Stylus does not completely solve task blocking, it offers simple heuristics to mitigate the issue.Task Diversity. Merging adapters into the base model overwrites the base model's prior distribution over an adapter's corresponding tasks. If an adapter is not finetuned on a diverse set of images, diversity is significantly reduced among different instances of the same task. We present three Retrieved Adapter Descriptions 42: This LoRA is for the concept of dragon, a mythical creature. It generates images of dragons with a variety of different appearances, including both Western and Eastern styles... 120: This LoRA steers the image generation towards a fantasy castle, with a focus on the building and its surroundings. The castle is depicted as a grand structure, often with towering spires, intricate architecture, and a sense of grandeur...This LoRA is designed to generate images of a Chinese dragon breathing fire. It generates images of a dragon with a long, serpentine body, covered in scales, with a large head and sharp teeth. The dragon is breathing fire, with flames coming out of its mouth... 1337: This LoRA is designed to generate images of animals breathing fire. It generates images of animals, such as rabbits, dragons, and frogs, breathing fire. The fire is shown as a bright, orange-yellow flame that is coming out of the animal's mouth...Provided above are the IDs and descriptions for different model adapters (e.g. LoRA) for Stable Diffusion that may be related to the prompt. Your goal is to fetch adapters that can improve image fidelity. The prompt is:Dragon breathing fire on a castle. % Chain-of-Thought Prompting First, segment the prompt into different tasks-concepts, styles, poses, celebrities, backgrounds, objects, actions, or adjectives-from the prompt's keywords.Here are the requirements for tasks:• Tasks should never introduce new information to the prompt. The topic must be selected from the prompt's keywords. • Different tasks must be orthogonal from each other. • All tasks combined must span the entirety of the prompt.• Prioritize choosing narrower tasks. You may merge tasks if a relevant adapter spans several tasks.Second, for each task, provide 0-5 of the most relevant model adapters to the task. For each adapter, infer an adapter's main function from its description. This function must directly match at least one task and the context of the prompt. If the adapter is indirectly relevant, do not include it.Here are the requirements for adapters:• Adapters should only be used at most once across all tasks. If an adapter is used in one task, it should not be used in another task. • Adapters should not introduce novel concepts or biases to the topic or the prompt. Do not include such adapters.• Adapters cannot encompass a broader scope relative to its assigned task. For example, if the task is about a "dog", the adapter cannot be about general "animals". • Adapters cannot be too narrow in scope relative to its assigned task. For example, if a task is about pandas, do not choose highly specific pandas such as the character "Po" from Kung Fu Panda. However, it is acceptable to choose adapters that modify the style of the task, such as "Red Pandas". • If an adapter spans multiple tasks, merge these tasks together. For example, if there is an adapter that is about "fluffy cats", merge the topics "fluffy" and "cats" together. • Avoid choosing NSFW and anthropormorphic adapters.Finally, for each selected adapter, provide a strong reason for why this adapter is relevant to the prompt, directly matches the keyword, and improves image quality.Give me the answer only. Please format your output as follows:Example 1: [Dictionary of tasks to the associated adapter ids and reasons for their selection.] Example 2:[ Dictionary of tasks to the associated adapter ids and reasons for their selection.] examples in Fig. 14c , over different prompts that specify multiple instances of the same task (teddy bears, women, and apples). We observe that all instances of each task are highly identical with one another. Stylus offers no solution to address or mitigate this problem.Low quality adapters. Low quality adapters can significantly degrade the quality of image generation, as shown by corrupted images in Fig. 14d . This issue typically arises from poor training data or from fine-tuning the adapter for too many epochs. Stylus attempts to blacklist such adapters. However, our blacklist is not comprehensive, and as a result, Stylus may still occasionally select low-quality adapters.Retrieval Errors. Stylus's retrieval process involves three stages, each introducing potential errors that can compound in later stages. For instance, the refiner may return incorrect descriptions of an adapter's task, while the composer may classify the adapter into an incorrect task. We detail three examples in Figure 4 . Stylus selects an "okapi" (forest giraffe) LoRA, known for its distinctive zebralike appearance, causing the generated giraffes to adopt the okapi's skin texture. In the middle, Stylus selects a flowery vase LoRA, a misinterpretation of the prompt "orange flowers placed in a vase." On the right, the composer incorrectly chooses a human baby adapter for the prompt "a baby daikon radish System Prompt: You are a photoshop expert judging which image has better composition quality.Scoring: Compositional quality scores can be 2 (very high quality), 1 (visually aesthetic but has elements with distortion/missing features/extra features), 0 (low visual quality, issues with texture/blur/visual artifacts).Scoring: Alignment scores can be 2 (fully aligned), 1 (incorporates part of the theme but not all), 0 (not aligned).Composition can be broken down into three main aspects:• Clarity: If the image is blurry, poorly lit, or has poor composition (objects obstructing each other), it gets scores 0. in a tutu.", resulting in images of babies instead of daikons. Stylus includes an option to self-repair faulty composer outputs with multi-turn conversations, which can improve adapter selection.The full prompts to GPT-4V as a judge for textual alignment, visual fidelity, and image diversity are specified in Tables 3 and 4 .To distinguish the two images (or groups of images), the VLM exploits multi-turn prompting: assigns lower scores by judging if an image has disfigured parts. For instance, diffusion models have trouble accurately depicting a human hand, oftentimes generating extra fingers. Finally, the VLM's final score depends on the detail of image. We find that adapters are able to bring greater detail to certain concepts. For example, an elephant adapter generates elephants with much greater detail than that of the base model. However, we note that the VLM is not good at detecting subtleties in detail. Diversity. For each prompt, we generate five images each for Stylus and the Stable Diffusion checkpoint. These images are then assessed with a VLM (Visual Language Model, GPT-4V) judge, which rates and ranks them based on diversity. In Tab. 4, we measure diversity through two metrics. The first metric, theme interpretation, measures diversity based on the interpretation of the prompt, which is often under-specified. We find that different thematic interpretations improves model response due to non-ambiguity. The second metric measures diversity by the variance of focus across different subjects. We find that many prompts often under-specify which subject is the focus on the image.A.6 Additional Diversity Scores Fig. 15 decomposes dFID scores over the top 100 keywords in the PartiPrompts dataset. We highlight that the largest differences stem from concepts, appearances, attributes, or styles. For example, Stylus excels over concepts ranging from animals ("bears", "sloth", and 'squirrel') to objects ("microphone", "box", and "jacket"). Selected attributes can include but are not limited to: ("white", "blue", and "photographic"). Regardless of keyword, Stylus attains higher diversity scores across the board.We acknowledge this work suffers from the same weakness other public domain image generation tools have with improper use for misinformation, producing explicit content, and reproducing copyrighted material from the training data. We strongly discourage the use of Stylus for these purposes and have taken preemptive measures to filter out potentially problematic adapters using Gemini. Further, Stylus is not meant to be used in production as proper guardrails are necessary for avoiding known gender and racial biases in the generated content. On release, we welcome community members to report problematic adapters missed in our initial curation of StylusDocs for removal from StylusDocs.an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations.Sourced from https://civitai.com/[28].We note that a large set of author descriptions are inaccurate, misleading, or absent. The refiner helped correct for human errors by using generated images as the ground truth, significantly improving our system.The debias prompts are "realistic, high quality" for Realistic-Vision-v6 and "anime style, high quality" for Counterfeit-v3, respectively.FID fails to disentangle image fidelity from diversity[32,41]. 


Decision: This paper explores systematically using adapters to improve the quality of diffusion model generated images, at reduced cost and increased efficiency. The paper studies how to automatically select adapters from a large library based on the given prompt.

The reviewers were all enthusiastic about the paper, and it would have a broad appeal to the NeurIPS audience. I therefore recommend an oral.

Reviews: review: paper_summary: paper_summary: This paper works on post-training optimization for image generation with stable diffusion models. They proposed three stages, refiner, retriever and composer, to personalize a SD model for the prompt and thus to generate the perfect images. The experimental result indicate the potential of the proposed method.
strength_weakness: strength_weakness: 1. Efficiency. In the paper, the authors compares the Stylus with the typical SD checkpoint, then the efficiency of stylus is not comparable to SD in terms of memory, CPU, GPU resources.
2. Fairness. The paper claim advantages over typical SD in terms of diversity and quality. While it is not quite fair. The Stylus has a model personalization process (from the post-training optimization pipelines), while the SD is using a static model checkpoint.
questions: questions: 1. Does the description (or the optimized version) can well represent the adapter?
2. in Eq.2, why do you directly use betha=0.8? If the 2nd term's value is much bigger than W_base, how do you deal with it?
limitations: limitations: N/A
paper_summary: paper_summary: The paper proposes Stylus, an approach for automatically selecting and combining fine-tuned adapters on particular tasks to improve the quality of image generation given a prompt. To evaluate Stylus, the paper introduces StylusDocs, a curated dataset containing 75K adapters with pre-computed adapter embeddings. Both the qualitative and quantitative results show that the proposed method outperforms Stable Diffusion and other retrieval methods.
strength_weakness: strength_weakness: - Some details about the proposed method are missing, making it hard to reproduce. In particular, sections 3.3 and 3.4 about the composer and the masking are not very clear. How are the adapters selected? How is masking applied?
questions: questions: No questions beyond the one in the weaknesses.
limitations: limitations: The paper does not have any obvious limitations that were not discussed.
paper_summary: paper_summary: The paper addresses the challenge of selecting and composing relevant adapters for generating high-fidelity images with diffusion models. Stylus introduces a three-stage process: refining adapter descriptions, retrieving relevant adapters based on user prompts, and composing them to create the final image. The paper highlights the development of StylusDocs, a dataset featuring 75K adapters with pre-computed embeddings. Evaluation results show that Stylus outperforms baseline models, achieving higher visual fidelity, textual alignment, and image diversity. The system is efficient and suitable for various image-to-image tasks, including translation and inpainting, demonstrating its versatility and effectiveness in improving image generation.
strength_weakness: strength_weakness: None. While adapters have been used in the past to improve image generation, this paper provides a much more coherent strategy to integrate them into VLMs.
questions: questions: None.
limitations: limitations: None. Great work, Authors!
paper_summary: paper_summary: This work presents a novel system, Stylus, designed to enhance the efficiency and effectiveness of generating high-quality images using diffusion models like Stable Diffusion. The key challenge addressed by this work is the automatic selection and composition of relevant fine-tuned adapters from a vast pool of over 100,000 available adapters, which are often poorly documented and highly customized. This work advances the field by providing a robust and automated solution by leveraging the vast number of available adapters.
strength_weakness: strength_weakness: - Unclear Motivation for the Method: In the refiner step, the paper does not clearly explain why Gemini Ultra is trusted to generate better adapter descriptions. If other multimodal language models (MLLMs) were used, how would the results differ?
- Incomplete Ablation Study: The ablation study is not comprehensive as it does not include an ablation of the refiner component. Understanding the impact of the refiner step on the overall performance of Stylus is crucial. 
- Quality Assurance of Adapter Descriptions: The paper does not provide sufficient details on how the quality of the adapter descriptions generated by the refiner is ensured. It is unclear whether any validation or verification steps were taken to confirm the accuracy and reliability of these descriptions. 
- Insufficient Description of Masking Process: The description of the masking process is not detailed enough. Specifically, the meaning of α_j in Equation 2 and the function Mask() are not adequately explained. Additionally, the masking process is not reflected in Figure 2, which outlines the Stylus algorithm.
questions: questions: - Motivation of the selected MLLM
- Additional ablation study
- Check the quality of adapter descriptions
- Details on masking process
limitations: limitations: Maybe potential bias and fairness issues will related with this work

Comments: title: If you have any further questions for the authors, please ask them now
All of my concerns have been addressed
comment: Thanks! We appreciate your enthusiastic review and constructive feedback!
Thank you for your response. My concerns have been addressed and I will keep my initial score.
We are glad that our rebuttal have addressed all your concerns and thank the reviewer for increasing the rating!
Dear reviewers,

Thank you for your continued efforts in helping create the best possible NeurIPS 2024.

These reviews are very enthusiastic; I urge you still to check each others reviews and the rebuttals to check whether you have any further questions for the authors -- you have one more day to interact with them.

Your AC
Thanks for the clarifications. All of my concerns have been addressed. I am raising the rating from weak accept to accept.
Thanks! We will definitely revise the description of the masking process in the manuscript to reflect the clarification we provided in the rebuttal.
Thank you for your response. Most of my concerns have been addressed. However, I suggest adding more details about the masking process to the manuscript for clarity. I'm pleased to raise my score.