Stylus: Automatic Adapter Selection for Diffusion Models
Authors: Michael Luo
Justin Wong
Brandon Trabucco
Yanping Huang
Joseph E. Gonzalez
Zhifeng Chen
Russ Salakhutdinov
Ion Stoica
Venue: NeurIPS 2024 oral (2024)
Keywords: Stable Diffusion
Diffusion-based Models
Computer Vision
Artificial Intelligence
RAG
Retrieval
Adapters
LoRA
Primary area: diffusion_based_models
TL;DR: Stylus automatically selects and composes adapters from a vast database of adapters for diffusion-based models.

Abstract: Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters—most of which are highly customized with insufficient descriptions. To generate high quality images, this paper explores the problem of matching the prompt to a Stylus of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP/FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model.

Stylus: Automatic Adapter Selection for Diffusion Models
Michael Luo'
Justin Wong'
Brandon Trabucco² Yanping Huang²
Joseph E. Gonzalez²
Zhifeng Chen Ruslan Salakhutdinov lon Stoica
UC Berkeley CCU MLD 'Google Deepmind
{mi chael. luo, wong wong- (btrabucc rsalakhu) @cs cmu, edu
{huangyp, zhifengc)@googleee
Stylus
SD
v1.5
Wooden dich racis 00 Countce holding auates SaUUOEEs, bencl Rooge and glossco.
umbralla
with elesbante
computer room with momitton
clift
AS6 kesboard and aundin
Iamap.
Figure 1: Adapter Selection. Given user-provided prompt, our method identifies highly relevant adapters (e.g. Low-Rank Adaptation, LoRA) that are closely aligned with the prompt's 's context and at least one of the prompt's keywords. Composing relevant adapters into Stable Diffusion improves visual fidelity, image diversity. and textual alignment. Note that these prompts sampled from MS-COCO [22].
Abstract
Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters- most of which are highly customized with insufficient descriptions. To generate high quality images, This paper explores the problem of matching the prompt to set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, developed StylusDocs, curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP/FII Pareto efficiency and is twice preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion. titisddfffiioongihuubb github..o io for more.
Introduction
In the evolving field of generative image models, finetuned adapters [7, 11] have become the standard, enabling custom image creation with reduced storage requirements. This shift has spurred the growth of extensive open-source platforms that encourage communities develop and share different
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Refirer
Composer
belox lis
LabA
VIA
Eneother
style
Adapter DB
Viette TIR
the plw
Retaaever
Secine for 2013 bepeerd, Alverity which
Iwo Bogs playing the
Eascrdle
Suacter
cancet:
LESA IELA 200n
Low penerttes vidreniitty
"Dags
4242
Figure 2: Stylus algorithm. Stylus consists of three stages. The refiner plugs adapter's 's model card through a VLM to generate textual descriptions of an adapter's 's task and then through an encoder to produce the corresponding text embedding. The retriever fetches candidate adapters that are relevant to the entire user prompt. Finally, the composer prunes and jointly categorizes the remaining adapters based on the prompt's ss tasks, which correspond to set of keywords.
adapters and model checkpoints, fueling the proliferation of creative AI art [28, 51]. As the ecosystem expands, the number of adapters has grown to over 100K, with Low-Rank Adaptation (LoRA) [14] emerging the dominant finetuning approach (see Fig. 3). A new paradigm has emerged where users manually select and creatively compose multiple adapters, on top of existing checkpoints, to generate high-fidelity images, moving beyond the standard approach of improving model class scale.
In light of performance gains, our paper explores the automatic selection of adapters based on user-provided prompts (see Fig. 1). However, selecting relevant adapters presents unique challenges compared to existing retrieval-based systems, which rank relevant texts via lookup embeddings [18]. Specifically, efficiently retrieving adapters requires converting adapters into lookup embeddings, a step made difficult with low-quality documentation or no direct access to training data-a common issue on open-source platforms. Furthermore, in the context of image generation, user prompts often imply multiple highly-specific tasks. For instance, the prompt "two dogs playing the snow" suggests that there are two tasks: generating images of "dogs" and "snow". This necessitates segmenting the prompt into various tasks (i.e. keywords) and selecting relevant adapters for each task, requirement beyond the scope of existing retrieval- based systems [9]. Finally, composing multiple adapters can degrade im- age quality, override existing concepts, and introduce unwanted biases into the model (see App. A.4).
We propose Stylus, a system that efficiently assesses user prompts to retrieve and compose sets of highly-relevant adapters, automatically augmenting generative models to produce diverse sets of high quality images. Stylus employs three-stage framework to address the above challenges. As shown in Fig. 2, the refiner plugs in an adapter's model card, including generated images and prompts, through a multi-modal vision-language model (VLM) and a text encoder to pre-compute concise adapter descriptions as lookup embeddings. Similar to prior retrieval methods [18], the relriever scores the relevance of each embedding against the user's entire prompt to retrieve a set of candidate adapters. Finally, the composer segments the prompt into disjoint tasks, further prunes irrelevant candidate adapters, and assigns the remaining adapters to each task. We show that the composer identifies highly-relevant adapters and avoids conceptually-simiine detrimental to image generation ($ 4.3). Finally, Stylus applies a binary mask to control the number of adapters per task, ensuring high image diversity by using different adapters for each image and mitigating challenges with composing many adapters.
adapters that introduce biases
To evaluate system, we introduce StylusDocs, an adapter dataset consisting of 75K LoRAs!. that contains pre-computed adapter documentations and embeddings from Stylus's refiner. Our results demonstrate that Stylus improves visual fidelity, textual alignment, and image diversity over popular Stable Diffuusion (SD 1.5) checkpoints-shiftine efficiency and achieving up 2x higher preference scores with humans and vision-language models (VLMs) as evaluators. As a system, Stylus is practical and does not present large overheads to the batch image generation process. Finally, Stylus can extend different mage-to-image application domains, such as image inpainting and translation.
the CLIP-FID Pareto curve towards greater
'Soureed from https://civital. [28].

---

2 Related Works
Adapters. Adapters efficiently fine-tune models on specific tasks with minimal parameter changes, reducing computational and storage requirements while maintaining similar performance to full fine-tuning [7, 11, 14].
Our study focuses on retrieving and merg- ing multiple Low-Rank adapters (LoRA), the popular approach within existing open-source communities [28, 29, 51J.
LoRA
Textual Inversion
Hypernswwork
100000
Adgpters (0)
75000
Adapter composition has emerged cial mechanism for enhancing the capabili- ties of foundational models across various applications [19, 36, 40, 45, 46]. For large language models (LLM), the linear com- bination of multiple adapters improves in-domain performance and cross-task generalization [3, 15, 16, 48, 49, 55]. In the image domain, merging LoRAs effec- tively composes different tasks casks-concepts, characters, poses, actions, and styles-together yiclding images of high fidelity that closely align with user specifications [25, 56]. Adapters also play a key role in synthetic data methods in few-shot computer vision [47]. Our approach advances this further by actively segmenting user prompts into distinct tasks and merging the appropriate adapters for each task.
50000
0
Civit Al
Hugging Face
Figure Number of Adapters. Civit Al boasts 100K+ adapters for Stable Diffusion, outpacing that of Hugging Face. Low-Rank Adaptation (LoRA) is the dominant approach for finetuning.
Retrieval-based Methods. Retrieval-based methods, such as retrieval -aggoveede generation (RAG), significantly improve model responses by adding semantically similar texts from vast external database [18]. These methods convert text to vector embeddings using text encoders, which are then ranked against a user prompt based on similarity metrics [4, 9, 21, 27, 37, 39]. Similarly, our work draws inspiration from RAG to encode adapters as vector embedings: leveraging visual-language foundational models (VLM) to generate semantic descriptions of adapters, which are then translated into embeddings.
A core limitation to RAG is limited precision, retrieving semi-relevant documents that do not exactly answer the prompt. This leads to a senelle-i--th-hhaaee the-haystack" problem, where more relevant documents are buried further down the list [9]. Recent work introduce reranking step; this technique uses cross-encoders to assess both the raw user prompt and the ranked set of raw texts individually, thereby discovering texts based actual relevance [27, 38). Rerankers have been successfully integrated with various LLM -apapliicattoon frameworks [2, 24, 35].
3 Our Method: Stylus
Adapter selection presents three distinct challenges compared to existing methods for retrieving text documents, as outlined in Section 2. First, computing embeddings for adapters is novel task, made more difficult without access to training datasets. Furthermore, the context of image generation, user prompts often specify multiple highly fine-grained tasks. This challenge extends beyond retrieving relevant adapters relative to the entire user prompt, but also matching them with specific tasks within the prompt. Finally, composing multiple adapters can degrade image quality and inject foreign biases into the model. Our three-stage framework below Refine, Retrieve, and Compose addressssddessss the above challenges (Fig. 2).
3.1 Refiner
The refiner is a two-stage pipeline designed to generate textual descriptions of an adapter's task and the corresponding text embeddings for retrieval purposes. This approach is analagous to pre-computed embeddings over an external database of texts in retrieval-based methods [18].
Given an adapter A,, the first stage a lasion-language model (VLM) that takes in the adapter's model card-a a set of randomly sampled example images from the model card I4 E {141, L:2,..., the cor- responding prompts P, E [P11, Prz, ...), and an author providdd deseription, 2 D, and returns an im- proved description D Optionally, the VLM also recommends the weight for LoRA-based adapters,
WWe note that a large set of author descriptions are inaccurate, misleading, or absent. The refiner helped correct for human errors by using generated images the ground truth, significantly improving our system.
3

---

Realistic- Vision-v8
Counterfeit feitefeiit-
Stylon
8D v1.5
Figure 4: Qualitative comparison between Stylus over realistic (left) and cartoon (right) style Stable Diffusion checkpoints. Stylus produces highly detailed images that correctly depicts keywords in the context of the prompt. For the prompt "A graffiti of corgi on the wall", our method correctly depicts a spray-painted corgi, whereas the checkpoint generates a realistic dog.
as the adapter weight is usually specified either in the author's description D; or the set of prompts P; a feature present in popular image generation software [1]. We denote this weight/coeffiiiett as 0g If information cannot be found, the LoRA's weight is set to og = 0.8. In our experiments, these improved descriptions were generated by Gemini Ultra [43] (see & A. for prompt). We chose the Gemini class of models since it has mature safety guardrailing. Specifically, Google's VertexAI API provides stringent safety settings block explicit content for the imput prompt. Safety filters helped us filter out around 30% of original adapters that were tagged as non- non-explicit by other model repositories.
The second stage uses an embedding model (E) to embed the text description D; for each adapters to yield embeddings, e; = E(D"). In our experiments, we create embeddings from OpenAl's eexedembb -umbedding---large 3-large model [21, 30]. We store pre-computed embeddings in a vector database, formally notated by the matrix, V.
3.2 Retriever
The retriever fetches the most relevant adapters over the entirety of the user's prompt using cosine similarity. Precisely, the retriever employs the same embedding model (E) to process the user prompt, q. generating embedding eq = Eg). Using the vector database, we calculate exact cosine similarity scores between the prompt's embedding e, and the embedding of each adapter in the matrix V.
vFvv
The similarity vector, $q |- scores the adapter descriptions by similarity. The retriever simply returns indices of the top-k adapters A& = top-k(s,), In our experiments, we find = 150 is effective for StylusDocs. We denote the set of i descriptions of the adapters, A as D
3.3 Composer
The composer serves a dual purpose: segmenting the prompt into tasks from a prompt's 's keywords and assigning retrieved adapters to tasks. This implicitly filters out adapters that are not semantically aligned with the prompt and detects those likely to introduce foreign bias to the prompt through keyword grounding. For example, if the prompt is "pandas eating bamboo", the composer may discard an irrelevant "grizzly bears" adapter and a biased "panda mascots" adapter.
The composer (C) is a function of the prompt (q), the top K adapters (Ak) from the retriever. Formally, denote the tasks identified by the composer as T(g) = {tt, tf..t J. The composer produces a mapping from task to adapters:
C(s, Ax) {(t,, Ak. It, T(q), Av, AK, Vj AR, Align(A,,
(1)
where Av, is the subset of adapters per task Li Align(Aj, 4:) is a predicate that holds if the adapter, Ay is aligned with the task, t;.
While the composer can be further improved by fine-tuning with human-labeled data [34], we find that prompting a long-context Large Language Model (LLM) suffices. The LLM accepts the adapter descriptions and the prompt as part of its context and returns a mapping of tasks to curated set of adapters. In practice, the alignment function is determined in the LLM's ' chain-of-thouglee procedure before it outputs the final mapping of udapters to tasks. In our implementation. we choose Gemini 1.5, with a 128K context window, as the composer's L.LM (see App. A.3 for the full prompt).
Stylus's 's composer is similar to reranking. Rerankers employ cross encoders (F) that compare the retriever's individual adapter descriptions, generated from the refiner, against the user prompt to determine better similarity scores: F(p D*). This prunes for adapters based on semantic relevance, thereby improving search quality, but not over keyword alignment. Our experimental ablations ($ 4.3) show that our composer outperforms existing rerankers (Cohere, rerank- wereeennglilhhhhh 0) [38].

---

DealiatiiVisiion COCO
RealistiVisien vó PartiPrompis
Counterfeii v3 /COC
Counterieu v3 Partirocapts
Stdue SD
Proformso Ruts
12.0%s
51%
25.0%
41,47%
31
Figure 5: Human Evaluation. Stylus achieves higher preference scores (2:1) over different datasets and Stable Diffusion checkpoints.
3.4 Masking
The composer maps tasks to corresponding sets of highly relevant adapters. To further mitigate sen- sitivity to low-quality adapters, Stylus reduces the number of selected adapters with straightforward masking scheme. Specifically, for each task, candidate masks are generated, and one is randomly se- lected to be applied over the set adapters. Formally, for given task, m, E {0, 1 A>lAul, is either one hot encoding, I, or Ô. forming set of possible masks, M₁. Across all tasks, masks are combined by taking the cross- cross-produs product, G M, x .22 x.. Mn. The combinatorial sets of masking schemes enable diverse linear combinations of adapters for single prompt. leading to highly-diverse images ($ 4.2.3). This approach also curtails the number of final adapters merged into the base model, minimizing the risk of composing low-quality adapters that may introduce undesirable effects to the image [56].
3.5 Merging
Stylus employs two key insights for effectively merging adapter weights. First, when applied to single task, large adapter weights can introduce notable visual artifacts, such over-saturation (Fig. 14a). Second, across multiple tasks, adapters tend to be orthogonal in the weight space, as they are designed to modify distinct, orthogonal concepts [8]. Hence, Stylus computes the final adapter weights by averaging weights per task and summing weights across tasks. This approach ensures that the adapter weights per task remain appropriately scaled.
We mathematically illustrate our merging scheme below. Recall, the refiner outputs a,, the recommended weight/coefficient, for each adapter. ($ 3.1). As shown in recent work [56], multiple LoRAs can be merged with the base model weights (Waase). We arrive at our final merged model weights by summing the adapter weights normalized by task. For mapping, {(t1, As,) ), (tz, Ak,).... (f., As.. }}, and g = (m,, m2... m,,) G, the final model weight is:
W = Woase + 3. D a/4;/1>
iS
where x; . MMsk((mm. A.) and A, is the LoRA's weight. We set 3 0.8 to mitigate image satura- tion, where assigning high adapter weights to an individual task (or concept) leads to sharp decreases in image quality (see App. A.4). For batch inference, Stylus returns images sorted by CLIP score.
Results
4.1 Experimental Setup
Adapter Testbed. Adapter selection requires large database of adapters to properly evaluate its performance. However, existing methods [15, 55] only evaluate against 50-350 adapters for language-basck tasks, which is insufficient for our use case, since image generation relies on highly fine grained tasks that span across many concepts, poses, styles, and characters. To bridge this gap, introduce StylusDocs, a comprehensive dataset that pulls 75K LoRAs from popular model repositories, Civit AI and HuggingFace [28, 51]. This dataset contains precomputed OpenAI embeddings [21] and improved adapter descriptions from Gemini Ultra- Vision [43], the output of Stylus's 's refiner component ($ 3.1). We further characterize the distribution of adapters in App. A.3.
Generation Details. We assess Stylus against Stable- Difffusionvv....5 [40] as the baseline model. Across experiments, we employ two well-known checkpoints: Realistic-Vistt Sisiii--iiiionv,, which excels in producing realistic images, and Counterfeit- which generates cartoon and anime-style images. Our image generation process integrates directly with Stable-Diffusion WebUI [1] and defaults to 35 denoising steps using the default DPM Solver++ scheduler [26). To replicate high-quality images from existing users, enable high-resolution upscaling generate 1024x1024
5

---

24
FID Score (10K)
CLIP (A)
FID (A)
Stylus
27.25 (+0.03)
22,05 (-1.1))
Reranker
25.48 (-1.74)
22.81 (-1.15)
Retriever-only
24.93 (-2.29)
24.68 (40.72)
SD.1.5
Random
26.34 (-0.88)
24.39 (+0.43)
Scyles
SD v1.5
27.22
23.96
503
259
CLIP Score (VTE-L/14)
(b) CLIP/FID scores and deltas over different retrieval methods (with CFG=6).
(a) Clip/FID Parcto Curve for COCO.
Figure 6: Automatic Evaluation Metrics, Figure (a) plots the CLIP/FID pareto curve. We observe Stylus shifts the curve down (improved visual fidelity, FID) and to the right (improved textual alignment, CLIP score) over a range of guidance values (CFG): [1. 1.5, 2, 3, 4, 6, 9, 12J. Table (b) evaluates Stylus against different retrieval methods. Stylus outperforms existing retrieval-base methods, attains the best FID score, and achieves similar CL.P score to Stable Diffusion.
Figure Image Diversity. Given same prompt. our method (left) generates more diverse comprehensive sets of images than that of cxisting Stable Diffusion checkpoints (right). Stylus's diversity comes from its masking scheme and the composer LLM's temperature paramcter.
from 512x512 images, with the default latent upscaler [17] and denoising strength set 0.7. For images generated by Stylus, we discovered adapters could shift the image style away from the checkpoint's original style. counteract this, we introduce a debias prompt injected the end of user prompt to steer images back to the checkpoint's style?. We launched 16 replicas of Stylus and Stable Diffusion on A100-800B GPUs for 4 weeks to generate images for evaluation.
4.2 Main Experiments
4.2.1 Human Evaluation.
To demonstrate our method's s general applicability, we evaluate Stylus over a cross product of two datasets. Microsoft COCO [22] and PartiPrompts [53], and two checkpoints, which generate realistic and anime-style images respectively. Examples of images generated in these styles are displayed in Figure 4; Stylus generates highly detailed images that better focus specific elements in the prompt.
To conduct human evaluation, we enlisted four users to assess 150 images from both Stylus and Stable Diffusion v1.5 for each dataset-checkpoint combination. These raters asked to indicate their preference for Stylus or Stable-Diffusion-vv.. ..5. In Fig. 5, users generally showed a preference for Stylus over existing model checkpoints. Although preference rates consistent across datasets, they varied significantly between different checkpoints. Adapters generally improve details their corresponding tasks (e.g. generate detailed elephants); however, for anime-style checkpoints, detail is less important, lowering preference scores.
4.2.2 Automatic Benchmarks.
We assess Stylus using two automatic benchmarks: CLIP [12], which measures the correlation between generated images' caption and users' prompts, and FID [13], which evaluates the diversity and aesthetic quality of image sets. We evaluate COCO 2014 validation dataset, with 10K
'The debias prompts are "realistic, high quality" for Realistic-Viste and "anime style, high quality" for Counterfeit-we respectively.

---

Lose
SD v1.5
Stylu
35
Preimence Win itan (N)
Normsizze dFTD
Photorenz Win Mote
40
19
25
Viaud Quality
Tostral Aligimmnt
GPT4V
JND
Prompe Length
(a) VLM Judge with GPT-4V
(b) Image Diversity (GPT- (c) Image diversity (dFID) versus increasing prompt lengths.
dFID)
Figure 8: Figure (a) and (b) evaluate the preference win rate using GPT-4V judge. Stylus achieves higher preference scores as judged by GPT-4V visual quality and image diversity. Figure (c) shows that Stylus achieves higher diversity scores than Stable Diffusion whens prompt length increases.
Stylus
Retrieeer
Random
A babss clesbunt underssech an 6dcct elephant.
huilding
greup jumping
kride holding atsanding.
yellow troin polling up to cement platform.
Taptop koybuseed
standing
Figure 9: Different Retrieval Methods. Stylus outperforms all other retrieval methods, which choose adapters than either introduce foreign concepts to the image or override other concepts the prompt, reducing textual alignment.
sampled prompts, and the Dealistic-vision--e checkpoint. Fig. 6a shows that Stylus shifts the Pareto curve towards greater efficiency, achieving better visual fidelity and textual alignment. This improvement aligns with our human evaluations, which suggest correlation between human preferences and the FID scores.
4.2.3 VLM as a Judge
We use VLM as a Judge to assess two key metrics: textual alignment and visual fidelity, simulating subjective assessments [5]. For visual fidelity, the VLM scores based on disfigured limbs and unrealistic composition of objects. When asked to make subjective judgements, autoregressive models tend exhibit bias towards the first option presented. To combat this, we evaluate Stylus under both orderings and only consider judgements that are consistent across reorderings; otherwise, we label a tie. In Fig. 8a, we assess evaluate 100 randomly sampled prompts from the PartiPrompts dataset [53]. Barring ties, we find visual fidelity achieves 60% win rate between Stylus and the Stable Diffusion realistic checkpoint, which conclusively consistent with the 68% win rate from our human evaluation. For textual alignment, we find negligible differences between Stylus and the Stable Diffusion checkpoint. As most prompts lead to tie, this indicates Stylus does introduce additional artifacts. We provide the full prompt in Appendix A.5.
7

---

4.2.4 Diversity per Prompt
Given identical prompts, Stylus generates highly diverse images due to different composer outputs and masking schemes. Qualitatively, Fig. 7 shows that Stylus generates dragons, maidens, and kitchens in diverse positions, concepts, and styles. To quantitatively assess this diversity, we use two metries:
Previous evaluations with FID [13] show that Stylus improves image quality and diversity across prompts". We define dFID specifically to evaluate diversity per prompt, calculated as the variance of latent embeddings from Inception [42]. Mathematically, dFID involves fitting Normal distribution N ((, Σ) to the latent features of InceptionV3, with the metric given by the trace of the covariance matrix, dFID Tr E.
dFID:
GPT-4V: We use VLM as a Judge to assess image diversity between images generated using Stylus and the Stable Diffusion checkpoint over PartiPrompts. Five images are sampled per group, Stylus and SD v1.5, with group positions randomly swapped across runs to avoid GPT-4V's positional bias [56]. Similar to VisDiff, ask GPT-4V to rate on scale from 0-2, where 0 indicates no diversity and 2 indicates high diversity [6]. Full prompt and additional details are provided in App A.5.
Fig. 8b displays preference rates and defines win when Stylus achieves higher dFID receives higher score from GPT-4V for a given prompt. Across 200 prompts, Stylus prevails in approximately 60% and 58% cases for dFID and GPT-4V respectively, excluding ties. Figure 8c compares Stylus with base Stable Diffusion 1.5 across prompt lengths, revealing that Stylus consistently produces more diverse images. Additional results measuring diversity per keyword are presented in Appendix A.6.
4.3 Ablations
4.3.1 Impact of Refiner
Table 10 evaluates the impact of different refiner pipelines on Stylus's end-to-end performance. Below, we describe each refiner baseline:
CLIP (A)
FID (A)
No-Refiner Gemini-Ultra Refiner
24.91 (-2.31)
24.26 (+0.30)
27.25 (+0.03)
22.05 (-1.91)
GPT--4o Refiner
28.04 (+0.82)
21.96 (-2.00)
No-Refiner: Stylus uses baseline adapter descriptions sourced from popular repositories such as Hugging- Face [28, 51]. These descriptions often low-quality and underspecified. Hence, Stylus chooses the wrong adapters and attains lower CLIP and FID scores relative SDv1.5.
SD v1.5
27.22
23.96
Figure 10: Refiner's impact on End2End performance. Without refiner, Stylus performs worse than SD) v1.5 due to the poor quality of author-provided descriptions. Annotating adapters with GPT- 4o significantly improves adapter descriptions and achieves higher CLIP/FID scores than Stylus's 's default refiner VLM, Gemini- Ultra.
This refiner, used throughout all our experiments, employs Gemini-Ultra to auto-generate enhanced adapter descriptions, improving both relevance and specificity. Consequently, Stylus attains better CLIP and FID scores than SDv1.5.
Gemini- Ultra Refiner:
The GPT-4o refiner, OpenAl's most advanced model, outputs the best adapter descriptions, yielding the highest performance gains across CLIP and FID scores. This baseline demonstrates that Stylus's ' end-to-end performance is highly dependent on the quality and specificity of adapter descriptions.
GPT-4o Refiner:
4.3.2 Alternative Retrieval- Methods
We benchmark Stylus's performance relative to different retrieval methods. For all baselines below, we select the top three adapters and merge them into the base model.
Random: Adapters are randomly sampled without replacement from StylusDocs.
Retriever: The retriever emulates standard RAG pipelines [18, 55], functionally equivalent to Stylus without the composer stage. Top adapters are fetched via cosine similarity over adapter embeddings.
Reranker: An alternative to Stylus's composer, the reranker fetches the retriever's 's adapters and plugs a cross-encoder that outputs the semantic similarity between adapters' descriptions and the prompt. We evaluate with Cohere's 's reranker endpoint [38].
As shown in Tab. 6b, Stylus achieves the highest CLIP and FID scores, outperforming all other baselines which fall behind the base Stable Diffusion model. First, both the retriever and reranker
*FID fails to disentangle image fidelity from diversity [32, 41].

---

Stylus
8:55
Retriever
SDv1.5
16
Composer
(BS-1)
Load Acapter
Image Geneestion
SDv1.5
(BS-2)
SDv1.5
(BS-4)
Time (6)
Figure 11: Comparison of Stylus's inference overheads with Stable Diffusion's inference time by batch size (BS), At BS=1, Stylus accounts for 75% of the image generation time, primarily due the composer processing long context prompts from adapter descriptions. However, Stylus's 's overhead decreases when batch size increases.
significantly underperform compared to Stable Diffusion. Each method selects adapters that are sim- ilar to the prompt but potentially introduce unrelated biases. In Fig. 9, both methods choose adapters related to elephant movie characters, which biases the concept of elephants and results in depictions of unrealistic elephants. Furthermore, both methods incorrectly assign weights to adapters, causing adapters' tasks to overshadow other tasks within the same prompt. In Fig. 9, both the reranker and retriever generate images solely focused on singular items tedss-bess, chairs, suitcases, or trains shins-while ignoring other elements specified in the prompt. We provide an analysis of failure modes in A.4.
Conversely. the random policy exhibits performance comparable, but slightly worse, to Stable Diffusion. The random baseline chooses adapters that are orthogonal to the user prompt. Thus, these adapters alter unrelated concepts, which does not affect image generation. In fact, we observed that the distribution of random policy's 's images in Fig. 9 were nearly identical to Stable Diffusion.
4.3.3 Breakdown of Stylus's Inference Time
This section breaks down the latency introduced by various components Stylus. We note that image generation time is independent of Stylus, as adapter weights are merged into the base model [14].
Figure 11 demonstrates the additional time Stylus contributes the image generation process across different batch sizes (BS), averaged over 100 randomly selected prompts. Specifically, Stylus adds 12.1 seconds the image generation time, with the composer accounting for 8.5 seconds. The composer's large overhead is due to long-contes prompts, which include adapter descriptions for the top 150 adapters and can reach up 20K+ tokens. Finally, when the BS is Stylus presents a 75% increase in overhead to the image generation process. However, Stylus's latency remains consistent across batch sizes, as the composer and retriever run only once. Hence, for batch inference workloads, Stylus incurs smaller overheads as batch size increases.
4.3.4 Image-Domain Tasks
Beyond text-to-image, Stylus applies across various image-to-image tasks. Fig. 12 demonstrates Stylus applied to two different image-to-image tasks: image translation and inpainting.
Image translation: Image translation involves transforming a source image into a variant image where the content remains unchanged, but the style is adapted to match the prompt's definition. Stylus effectively converts images into their target domains by selecting the appropriate LoRA, which provides higher fidelity description of the style. We present examples in Fig 12a. For a yellow motorcycle, Stylus identifies voxel LoRA that more effectively decomposes the motorcycle into discrete 3D bits. For natural landscape, Stylus successfully incorporates more volcanic elements, covering the landscape in magma.
Inpainting: Inpainting involves filling in missing data within designated region of an image, typically outlined by a binary mask. Stylus excels in accurately filling the masked regions with specific characters and themes, enhancing visual fidelity. We provide further examples in Fig. 12b, demonstrating how Stylus can precisely inpaint various celebrities and characters (left), as well as effectively introduce new styles to a rabbit (right).

---

8iska
50 v1.
Barm
Fivey mi
Vesil Stple
Pruet Alinahs
Ta Oogh Style
lave
(a) Image Translation. Stylus chooses relevant adapters that better adapt new styles and elements into existing images.
Bource
vs..
8D vt.5
Jaszece
Reppes
Freezon
Glaas Dasas
Buage
Bibet Bunay
(b) Inpainting. Stylus chooses adapters than can better introduce new characters concepts into the inpainted mask.
Figure 12: Stylus over different image-to-image tasks.
Discussion
The strategic composition and routing of adapters Stylus introduce a new dimension of model performance, broadening the scope of potential applications. One such application is the automatic creation of agentic workflows [54, 57]. For instance. Stylus's composer can decompose complex task into graph of subtasks and assign them to specialized agents to improve end-to-end performance. Additionally, routing can extend beyond adapters encompass different models, allowing Stylus to optimize the fost-performance tradeoff by dynamically selecting between high- righ-performing. resource-ines intensive models and more efficient, lower-cost models [31, 33]. Finally.for fact verification, adapters have shown significant potential in reducing hallucinations [10, 44]. Stylus can selectively use domain-specific, fine-tuned models to enhance factual accuracy and better verify claims.
As demonstrated in Sec. 4, Stylus demonstrates significant potential for improvement, adapter composition introduces future research challenges beyond the scope of this work. summary of Stylus's failure cases are provided in Fig. 14. Specifically, adapters can restrict certain concepts from appearing in image and limit diversity among multiple subjects within a scene. While Stylus does not fundamentally solve these challenges, Stylus reduces the likelihood of these problems occurring by reducing the number of adapters through its masking algorithm. Lastly, Stylus introduces noticeable overheads to the inference pipeline, primarily stemming from the composer's 's long context prompts, which can be accelerated with various sequence parallel techniques [20, 23].
Conclusion
We propose Stylus, flexible algorithm that automatically selects and composes adapters to generate better images. Our method leverages three-stage framework that precomputes adapters as lookup embeddings and retrieves most relevant adapters based on prompts' keywords. To evaluate Stylus, develop StylusDocs, a processed dataset featuring 75K adapters and pre-computed adapter embeddings. Our evaluation of Stylus, across automatic metrics, humans, and vision-language models, demonstrate that Stylus achieves better visual fidelity, textual alignment, and image diversity than existing Stable Diffusion checkpoints.
Acknowledgement
We thank Lisa Dunlap, Ansh Chaurasia, Siyuan Zhuang, Sijun Tan, Chris Douglas, Tianjun Zhang, and Shishir Patil for their insightful discussion. We thank Google Deepmind for funding this project, providing AI infrastructure, and provisioning Gemini endpoints. Sky Computing Lab is supported by gifts from Accenture, AMD, Anyscale, Google, IBM. Intel, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Samsung SDS, SAP, Uber, and VMware.
10

---

References
[1] AUTOMATICI111. Stable Diffusion Web UI, August 2022. 4,
[2] Harrison Chase. LangChain, October 2022.
[3] Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, and Jesse Dodge, Adaptersoup:
Weight averaging to improve generalization of pretrained language models, 2023.
[4] Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
[5] Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Š Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 30039-30069, Curran Associates, Inc., 2023.
[6] Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E. Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. Conference on Computer Vision and Puttern Recognition (CVPR). 2024. 8, 20
[7] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, 2022. 1,3
[8] Rohit Gandikota, Joanna Materzyúska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311. /2092, 2023.
[9] Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. aatrivaal-augmenter generation for large language models: A survey, 2024. 2.3
[10] Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning ilms on new knowledge encourage hallucinations?, 2024. 10
[11] David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016. 1,
[12] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. 2022. 6
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium, 2018. 6, 8
[14] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. 2,
[15] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang. Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024. 3.
[16] Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. Personalized soups: Personalized large language model alignment via post-hoc parameter merging, 2023.
[17] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models, 2022.
[18] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmenned generation for knowledge-intensive nlp tasks, 2021. 2, 3,
[19] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024.
[20] Bin Lin, Chen Zhang, Tao Peng, Hanyu Zhao, Wencong Xiao, Minmin Sun, Anmin Liu, Zhipeng Zhang, Lanbo Li, Xiafei Qiu, Shen Li, Zhigang Ji, Tao Xie, Yong Li, and Wei Lin. Infinite Mimiteellm: Efficient I1m service for long context with distattention and distributed kveache, 2024. 10
[21] Jimmy Lin, Ronak Pradeep, Tommaso Teofili, and Jasper Xian. Vector search with openai embeddings: Lucene is all you need, 2023. 3, 4,

---

[22] Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. 1, 1,6, 15
[23] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023. 10
[24] Jerry Liu. Llamalndex, 11 2022. 3
[25] Nan Liu, Yilun Du, Shuang Li, Joshua Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-imapee generative models, 2023. 3
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. DPM-solver++:
Fast solver for guided sampling of diffusion probabilistic models, 2023. 5
[27] Tengyu Ma. Vectorize your data to gear up your ai stack., 2023. 3
[28] Justin Maier. The home of open-source generative ai, 2022. 3, 15
[29] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-effiieet fine-tuning methods. 2022.
https://gitee thwbb.tttttuuut..//e com//uuggiggaace/peeeeeeooo
[30] Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. Sfr-embeddingg
wwwembbddingggmmtttrrrnaaaaaaa meharne text retrieval with transfer learning. Salesforce Al Research Blog, 2024.
[31] Alireza Mohammadshahi, Arshad Rafiq Shaikh, and Majid Yazdani. Routoo: Learning to route to large language models effectively, 2024. 10
[32] Muhammad Ferjad Naeem, Seong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Leurning Research, pages 7 7176-7185. PMLR, 2020.
[33] Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. Routellm: Learning to route Ilms with preference data,
2024. 10
[34] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. 4
[35] Malte Pietsch, Timo Möller, Bogdan Kostic, Julian Risch, Massimiliano Pippi, Mayank Jobanputra, Sara Zanzottera, Silvano Cerza, Vladimir Blagojevic, Thomas Stadelmann, Tanay Soni, and Sebastian Lec. Haystack: the end-to-end NLP framework for pragmatic builders, November 2019. 3
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 3
[38] Nils Reimers. Say goodbye to irrelevant search results: Cohere rerank is here, 2023. 3, 4,
[39] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2022. 3.
[41] Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Samy Bengio, Hanna M. Wallach,
12

---

Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 5234-5243 2018. 8
[42] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pultern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016,
pages 2818-2826. IEEE Computer Society, 2016. 8
[43] Gemini Team. Gemini: family of highly capable multimodal models, 2023. 4, 5, 15
[44] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023. 10
[45] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. 3
[46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale. Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. 3
[47] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models, 2023. 3
[48] Hanqing Wang. Bowen Ping, Shuo Wang, Xu Han, Yun Chen, Zhiyuan Liu, and Maosong Sun. Lora-flow: Dynamic lora fusion for large language models in generative tasks, 2024. 3
[49] Xinyi Wang, Yulia Tsvetkov, Sebastian Ruder, and Graham Neubig. Efficient time adapter ensembling for low-resource language varieties. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 730-737, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. 3
[50] Jason Wei, Xuezhi Wang. Dale Schuurmans. Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of -f-thuugtt prompting elicits reasoning in large language models, 2023. 15, 20
[51] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. 2, 3, 5, 8, 15
[52] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas I.. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
15, 20
[53] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation, 2022. 6, 7
13

---

[54] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang. Bingnan Zheng. Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation, 2024. 10
[55] Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. Loraretriever: Input-aware lora retrieval and composition for mixed tasks in the wild,
2024. 5,
[56]
Ming Zhong, Yelong Shen, Shuohang Wang, Yadong Lu, Yizhu Jiao, Siru Ouyang, Donghan Yu, Jiawei Han, and Weizhu Chen. Multi-lora composition for image generation, 2024. 3, 5,
[57] Mingehen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Language agents as optimizable graphs, 2024. 10
14

---

g1a
1.50
Calagety
Tap 503 Adaptors Downiied Count
Distribution of adapters calegories.
(b) Top 500 adapters ranked by (%) downloads.
Figure 13: Characterization of Civit Adapter in StylusDocs. (a) Most adapters are categorized as characters celebrities. (b) Adapter popularity exhibits power-law distribution, with the top adapters receiving exponentially more downloads than the others.
A Appendix
A.1 Details of the Refiner VLM
We provide complete example input to the refiner's VLM in Tab. 1. The prompt utilizes (CoT) prompting, which decomposes the VLM's goal of producing better adapter descriptions into two steps [50, 52]. Initially, the VLM categorizes the adapter's ss task into one of several topics-Soi such as concepts, styles, characters, or poses. Subsequently, the VLM is prompted to elaborate why the adapter is associated with particular topic and how it modifies images within that context. We found that this two step logical process significantly improved the structure and quality of model responses.
A.2 Details of the Composer LLM
We provide a full example prompt of the composer's LLM component in Tab. 2, which is plugged through the Gemini 1.5 endpoint [43]. Our experiments feed descriptions of the top 150 adapters into the LLM's 's context. Using a Chain-of Thoouhht (CoT) approach, the prompt is structured to first identify keywords or tasks, then allocate appropriate adapters to these tasks. If necessary, merges keywords for adapters that span multiple tasks [50, 52].
A.3 StylusDocs Characterization
This section describes StylusDocs, which comprises of 76K Low Rank Adapters (LoRAs) from public repositories, including Civit Al and Hugging Face [28, 51]. We excluded NSFW-labeled adapters from the Civit AI dataset, which originally contained over 100K LoRAs. Figure 13 illustrates the distribution of adapters across various semantic categories and their popularity, measured by download counts. A significant majority, 70%, of adapters belong to the character and celebrity category, primarily consisting of anime or game characters. Another 13% of adapters modify image style, 8% adjust clothing, and 4% represent various concepts (Fig. 13a). These statistics indicate that our experiments consider a minor proportion of adapters, as the COCO dataset does not feature characters or celebrities [22]. Despite this, Stylus outperforms base Stable Diffusion. Furthermore, the popularity of adapters follows a Pareto distribution, where the top adapters receive exponentially more downloads than the others (Fig. 13a). However, the top adapter accounts for only 0.5% of total downloads, which suggests that the distribution long-tailed.
A.4 Failure Modes
We detail different failure modes that were discovered while developing Stylus.
Image saturation. The quality of image generation is highly depend on adapters' weights. If the assigned weight is above the recommended value, the adapter negatively impacts image generation, leading to a growing number of visual inconsistencies and artifacts. In Fig. 14a, assigning a high weight to "James Bond" LoRA increases images exposure and introducing significant visual tearing. Stylus mitigates over-saturation with its refiner component, which extract the right adapter weights from the adapter's 's model card. Lastly, Stylus uniformly weights adapters based on their associated tasks, ensuring that similar adapters do not significantly impact their corresponding tasks.
Task Blocking. Composing adapters presents the risk of overwriting existing concepts or tasks specified in the prompt and other selected adapters. We illustrate several examples in Figure 2-a -
15

---

Image Prompts
Prompt 1: Photo of Dwayne Johnson, wearing military clothes and cap, dramatic lighting, <lora: : TheRockV3:0.0..
Prompt 2: Photo of Dwayne Johnson, wearing a Superman suit, high quality, <lora: TheRock33:12.
Prompt 3: Photo of Dwayne Johnson, wearing an Armani tuxedo, 2Looa:/hheRock3:00..99
Model Card Description
Title: Dwayne "The Rock" Johnson (LoRA)
Tags: Celebrity, Photorealistic, Hollywood, Celeb Trigger Words: Th3R0ck
Description: Had make this one, due to Kevin Hart Lora. Recommended lora strength: 0.9. % Author descriptions may be misleading or incomplete.
Your goal improve the description of model adapter's task Stable Diffusion, with images, prompts, and descriptions pulled from popular model repositories. Above, we have provided the following information and the associated constraints:
Examples of generated images (from left to right) from the adapter and corresponding user-provided prompts. Some prompts may specify the adapter weight (i.e. <lora: NAME: WEIGHT>). If provided, you will need to infer the adapter's name and weight. Prioritize this weight over the author's 's recommended weight.
2. The adapter's model card from the original author. This includes the title, tags, trigger words, and description.
The model card description may he incorrect, mislcading, or incomplete.
The model card may specify the weight of the model adapter, or the recommended range. Find the recommended weight of the adapter (default is 0.8).
% Chain-of-Thoughtt Prompting
Again, your mission is to provide clear description of the model's 's adapter purpose and its impact on the image. To do so, you should implicitly categorize the model adapter into only one of the following topics: [Concept, Style, Posc, Action, Celebrity/Carratter, Clothing, Background, Building, Vehicle, Animal, Action]. Do not associate adapter with topic that is vague or uninteresting.
First, describe the topic associated with the adapter and explain how this adapter alters the images, based the elements observed in the example images. Your requirements are:
Do not descrihe any training or datasct-related details.
Provide additional context from your prior knowledge if there is insufficient information.
Do hallucinate and repeat text. Output only english words and sentences.
Second, recommend an optimal weight for the adapter as a float. Do not specify a range, only give one value.
Please format your output as follows:
Example 1:
[Description of adapter and weight]
Example 2:
[Description of adapter and its weight)
Table 1: Full prompt for the refiner VL.M to generate hetter adapter descriptions.
train LoRA overrides the toy train concept (left), a park bench LoRA masks a person in an orange blanket (middle), and fancy cake LoRA erases the image of
eating the cake (right). Task blocking typically arises from two main issues: the adapter weight set too high or too many adapters merged into the base model. Stylus addresses this by reducing an adapter's weight with uniform weighting per task, while the masking scheme reduces the number of selected adapters. Although Stylus does not completely solve task blocking, offers simple heuristics to mitigate the issue.
Task Diversity. Merging adapters into the base model overwrites the base model's prior distribution over an adapter's corresponding tasks. an adapter not finetuned on a diverse set of images, diversity is significantly reduced among different instances of the same task. We present three
16

---

Retrieved Adapter Descriptions
42: This LoRA is for the concept of dragon, mythical creature. generates images dragons with variety of different appearances, including both Western and Fastern styles...
120: This LoRA steers the image generation towards a fantasy castle, with focus on the building and its surroundings. The castle is depicted as a grand structure, often with towering spires, intricate architecture, and a sense of grandeur...
This LoRA is designed to generate images of a Chinese dragon breathing fire. It generates images of dragon with a long, serpentine body, covered in scales, with large head and sharp tecth. The dragon is breathing fire, with flames coming out of its mouth...
3478:
1337: This LoRA is designed to generate images of animals breathing fire. It generates images of animals, such as rabbits, dragons, and frogs, breathing fire. The fire is shown as a bright, orange-yellow flame that is coming out of the animal's mouth...
Provided above are the IDs and descriptions for different model adapters (c.g. LoRA) for Stable Diffusion that may be related to the prompt. Your goal is to fetch adapters that can improve image fidelity. The prompt is:
Dragon breathing fire on a castle.
% Chain-of Thought Prompting
First, segment the prompt into different tasks concepts, styles, poses, celebrities, backgrounds, objects, actions, adjectives from the prompt's is keywords.
Here are the requirements for tasks:
Tasks should never introduce new information to the prompt. The topic must be selected from the prompt's
keywords.
Different tasks must be orthogonal from each other.
All tasks combined must span the entirety of the prompt.
Prioritize choosing narrower tasks. You may merge tasks if relevant adapter spans several tasks.
Second, for each task, provide 0-5 of the most relevant model adapters to the task. For each adapter. infer an adapter's main function from its description. This function must directly match at least one task and the context of the prompt. If the adapter is indirectly relevant, do not include it.
Here are the requirements for adapters:
Adapters should only be used at most once across all tasks. If an adapter is used in one task, it should not be
used another task.
Adapters should not introduce novel concepts or biases to the topic or the prompt. Do not include such adapters.
Adapters cannot encompass a broader scope relative to its assigned task. For example, if the task is about a "dog". the adapter cannot be about general "animals".
Adapters cannot he too narrow in scope relative its assigned task. For example, if task about pandas, do not choose highly specific pandas such as the character "Po" from Kung Fu Panda. However, it acceptable to choose adapters that modify the style of the task, such as "Red Pandas". If an adapter spans multiple tasks, merge these tasks together. For example, if there is an adapter that is about "fluffy cats", merge the topics "fluffy" and "cats" together. Avoid choosing NSFW and anthropormorphic adapters.
Finally, for each selected adapter, provide strong reason for why this adapter is relevant to the prompt, directly matches the keyword, and improves image quality.
Give me the answer only. Please format your output as follows:
Example 1:
[Dictionary of tasks to the associated adapter ids and reasons for their selection.
Example 2: Dictionary of tasks the associated adapter ids and reasons for their selection.)
Table 2: Full prompt for the composer L.I.M
17

---

W=0
w=0.5
w=1.0
w=1.5
w=2.0
(a) Image Saturation. Assigning too high of weight a "James Bond" adapter leads to significant degradation visual fidelity.
Stylus
Stylus
SD
SD
v1.5
(b) Task Blocking, Adapters black prompt's other adapter's tasks (i.e. toy trains, person in orange blanket, or man eating cake).
(c) Task Diversity. Adding adapter reduces diversity of instances within a single task (i.e. teddy bears, woman, and apples).
v1.5
(d) Low quality adapters. Low quality adapters can significantly impact visual fidelity. We blacklist such adapters.
Retrieval Errors. Retrieval emros lead foreign biases image generation and deliberate misinterpretations of the prompt.
Figure 14: Categorization of Different Failure Modes.
examples in Fig. 14c, over different prompts that specify multiple instances of the same task (teddy bears, women, and apples). We observe that all instances each task are highly identical with one another. Stylus offers solution to address or mitigate this problem.
Low quality adapters. Low quality adapters can significantly degrade the quality of image generation, as shown by corrupted images in Fig. 14d. This issue typically arises from poor training data or from fine-tuning the adapter for too many epochs. Stylus attempts to blacklist such adapters. However, our blacklist not comprehensive, and as result, Stylus may still occasionally select low-quality adapters.
Stylus's retrieval process involves three stages, each introducing potential errors that can compound in later stages. For instance, the refiner may return incorrect descriptions of an adapter's 's task, while the composer may classify the adapter into an incorrect task. We detail three examples Figure Stylus selects an "okapi" (forest giraffe) LoRA, known for its distinctive zebra- like appearance, causing the generated giraffes to adopt the okapi's 's skin texture. In the middle, Stylus selects flowery vase LoRA, misinterpretation of the prompt "orange flowers placed in vase." the right, the composer incorrectly chooses human baby adapter for the prompt "a baby daikon radish
Retrieval Errors.
18

---

System Prompt:
You are a photoshop expert judging which image has better composition quality.
Compositional quality scores can be (very high quality), (visually aesthetic but has elements with distortion/missing features/extra features), (low visual quality, issues with texture/blur/visue artifacts).
Scoring:
Scoring: Alignment scores be (fully aligned), 1 (in- corporates part of the theme but all), 0 (not aligned).
Composition can broken down into three main aspects:
We provide several examples:
the prompt is 'shoes', and an image is sock, this is not aligned and gets a score of O. If the prompt is 'shoes without laces', but the shoes have laces, this is somewhat aligned and gets score of 1.
Clarity: If the image is blurry, poorly lit,
has poor composition (objects obstructing each other), it gets scores O.
Disfigured Parts: This applies to both body parts of humans and animals well as objects like motoreycles. If the image has hand that has fingers it gets a for having otherwise nor- mal fingers, but the hand should not have two fingers. If the fingers themselves are disfigured showing lips and teeth warped in, it gets O. Detail: If
If the prompt is concert without fans', but there's 's fans in the image, pick the images that show fewer fans.
of a sailboat's 's sail shows dynamic ripples and ornate patterns, this shows detail and should get a score of 2. If it's monochrome and flat, it gets score of 1. If looks like cartoon and inconsistent with environment, give a score
User:
This IMAGE A. Reply' ACCK' % Generated Image from Group
Assistant: ACK
User:
This IMAGE B. Reply ACCK' % Generated Images from Group B
Assistant: ACK
User:
Rate the quality of the images in GROUP A and GROUP B. For cach image, provide score and explanation.
Image A Quality: -SCORE>
Image Quality: <SSOOEE Preference: Group Group SIup-=-CHOOCCSCCEELLAAAATOO
% Prevent VLM from returning neutral results.
make my own judgement using your results, your response just an opinion as part of rigorous process. provide additional requirements below:
You must pick group for 'Better Quality' / /Better Alignment", neither is not an option.
If it's close call, make choice first then explain why in parenthesis.
Table 3: Full prompt judging compositional quality (left) or textual alignment (right) using VLM.
in tutu.", resulting in images of babies instead of daikons. Stylus includes an option to self-repair faulty composer outputs with multi-turn conversations, which can improve adapter selection.
A.5
VLM as a Judge
The full prompts to GPT-4V as judge for textual alignment, visual fidelity, and image diversity specified in Tables 3 and 4.
To distinguish the two images (or groups of images), the VL.M exploits multi-turn prompting: We provide each image (or group of images) labeled with IMAGE/GROUP A or IMAGE/GROUP B. Note
19

---

System Prompt:
You are a photoshop expert judging which set of images more diverse.
Scoring: Diversity scores can be (very diverse), 1 (somewhat diverse), 0 (not diverse).
Diversity can be decomposed based 1) the interpretation of the theme and 2) the main subject.
Theme Interpretation: The theme can vary based on interpretation. The theme "it's raining cats and dogs" can have a literal interpretation as cats and dogs falling from the sky or figurative interpretation as heavy rain. The images are diverse, since they show both weather and animals. If the group only contains images of heavy rain or animals, diversity score of should be given.
Main Subject: The main subject changes based on the focus across different subjects. set of images that contains mix images apples and children dressed as different kinds apples is more diverse than a set
with only children dressed as apples. Note the more diverse set has children as the subject for some images and apples as the subject for other images.
User:
This GROUP A. Reply 'ACK'.
% Set of 5 Generated Images from Group
Assistant: ACK
User:
This is GROUP B. Reply 'ACK' % Set of 5 Generated Images from Group B
Assistant: ACK
User:
Rate the diversity of the images in GROUP A and GROUP B. For each group, provide a score and explanation.
Group A Diversity:
<SCORE>/
Group B Diversity: <SCOREE<CDXPPLNNAIOO Preference: Group <CHOICEX
% Prevent VLM from returning neutral results.
I'll make my own judgement using your results, your response just opinion part of rigorous process. provide additional requirements below:
Don't forget to reward different main subjects in the diversity score.
You must pick group for 'More Diversity," neither is not an option.
If it's close call, make choice first then explain why in parenthesis.
Table 4: Full prompt judging diversity using VL.M.
that the ACK messages provide the rubric, detailed instructions, reminders, and example model outputs in our prompt. For scoring, the VLM employs Chain-of-Thouuge (CoT) prompting to output scores 0-2, similar to VisD- iff [6. 50, 52]. We observe that larger ranges (5-10) leads the model towards abstaining from making decisions, it avoids outputting extreme scores. However, the score range 0-2 provides the VLM sufficient granularity to express preferences and prompt the model to summarize the key differences.
generated by the VLM; instead, it is part of VLM's context window. We
Textual Alignment. The VLM scores how well generated image follows the prompt's specifications. We note that prompts with negations (e.g. "concert with no fans" or "harbor with no boats"") fail for both Stylus and the Stable Diffusion checkpoint. Hence, we prompted the VLM to assign better scores for images that produced less fans or boats. Furthermore, as adapters can potentially block existing concepts in the image (see Fig. 14b), the VLM allocates partial credit in scenarios where images partially capture the set of keywords in the prompt.
Visual Quality. Our evaluation assesses visual quality through three metries: clarify, disfigurements, and detail. First, the VLM assigns low clarity scores if an image is blurry, poorly lighted, or exhibits poor compositional quality. We note that LoRAs are trained over specific tasks/concepts; the model determines how to compose different concepts. For instance, a rhinoceros LoRA combined with a motorcycle LoRA led to images of motorcycles draped with rhinoceros hide. As such, the VLM assigns partial credit when the model fails to combine concepts meaningful way. Second, the VLM
20

---

Parti Prompts Diversity
SD 1.5
Stytus
Figure 15: dFID for top 100 keywords in PartiPrompts dataset. Stylus leads to consistently higher diversity when compared to Stable Diffusion checkpoints, especially for words describing concepts and attributes.
assigns lower scores by judging an image has disfigured parts. For instance, diffusion models have trouble accurately depicting a human hand, oftentimes generating extra fingers. Finally, the VLM's final score depends on the detail of image. We find that adapters are able to bring greater detail to certain concepts. For example, elephant adapter generates elephants with much greater detail than that the base model. However, we note that the VLM is not good at detecting subtleties in detail.
Diversity. For each prompt, we generate five images each for Stylus and the Stable Diffusion check- point. These images are then assessed with a VLM (Visual Language Model, GPT-4V) judge, which rates and ranks them based diversity. In Tab. first metric, theme interpretation, measures diversity based on the interpretation of the prompt, which often under-specified. We find that different thematic interpretations improves model response due to non-ambiguity. The second metric measures diversity by the variance of focus across different subjects. We find that many prompts often under-specify which subject is the focus on the image.
measure diversity through two metries. The
A.6 Additional Diversity Scores
Fig. 15 decomposes dFID scores the top 100 keywords in the PartiPrompts dataset. We highlight that the largest differences stem from concepts, appearances, attributes, or styles. For example, Stylus excels over concepts ranging from animals ("bears", "sloth", and 'squirrel") objects ("microphone", "box", and "jacket"). Selected attributes can include but are not limited to: ("white", "blue", and "photographic"). Regardless of keyword, Stylus attains higher diversity scores across the board.
A.7 Disclaimer
We acknowledge this work suffers from the same weakness other public domain image generation tools have with improper use for misinformation. producing explicit content, and reproducing copy- righted material from the training data. We strongly discourage the use of Stylus for these purposes and have taken preemptive measures to filter out potentially problematic adapters using Gemini. Fur- ther. Stylus is not meant to be used in production proper guardrails necessary for avoiding known gender and racial biases the generated content. On release, we welcome community members to re- port problematic adapters missed in initial curation of StylusDocs for removal from StylusDocs.
21

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments claim the contribution of StylusDoca, a dataset 75K adapters with improved documentation generated by VLM. Stylus uses StylusDocs as part of the evaluation. Further, our claims that Stylus improves visual fidelity and image diversity are substantiated by the CLIP/FID pareto curve, human evaluators, and GPT-4V as a judge.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We address the limitations of Stylus in the Appendix A.4, which address the various possible failure cases that emerge from composing different adapters. We also address the potential overheads of Stylus's ' inference time in Sec. 4.3.3, which can be large for small batch sizes.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play
22

---

an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Stylus is not a theory paper and hence does not make theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Yes, our paper discloses the necessary information to reproduce the main results in our paper. We provide the full sample prompts to the Refiner VI.M and Composer L.M in the Appendix. We discuss which models (Gemini Ultra- Vision 1.5) were used to generate the adapter descriptions for StylusDocs and the composer's 's output adapters. We also concisely describe our experiments, hyperparameters, and datasets, providing the full sample evaluation prompts for VLM as judge in Tab. 4. Finally, Stylus is open-sourced. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
23

---

(c) If the contribution is new model (e.g., large language model), then there should either be a way to access this model for reproducing the results way reproduce the model (e.g., with an open-source dataset or instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Stylus is already open-source, with all experiments made available to individuals online on Github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https:
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines battps://nise http://iins..ccpplllpee wwtppppcc/gguueiieeeeeeeeepoeeee
http://ooooeeessseeeeeeeoo coPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Detalls
Question: Does the paper specify all the training and test details (e.g., data splits, hyperpa- rameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Our paper specifies the hyperparameters for both Stylus and the parameters for generating images from Stable Diffusion (such as denoising steps).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
report
Answer: [Yes]
24

---

Justification: In Figure 8, we show the one standard deviation as shaded region. We follow existing conventions for reporting the CLIP/FID pareto curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: We mention that course of several weeks Sec. 4.
experiments 8 A100-80GB GPUs over the
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g.. preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : //nnurruisp.../ wttppps.cccppppcpppeeepeppp httpr/ppp..pppppppt/attttaaaaaaa ines?
Answer: [Yes]
Justification: We only access public domain adapters and use Google's default safety filter for Gemini models to remove potentially harmful adapters. We also remove all adapters that were tagged as explicit from Civit AI. The quality our StylusDocs is carefully curated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make consideration due to laws or regulations in their jurisdiction).
to preserve anonymity (e.g., if there is special
25

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the conclusion, we note that StylusDocs is carefully curated to avoid explicit and low-quality adapters, but this may not guaranteed. Further, the work suffers from the same weakness other public domain image generation tools have with improper use and misinformation, including unintentionally producing explicit content and reproducing copyrighted material from the training data.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In App.A.7, we discuss the continual curation process by which we ask com- munity members to report problematic adapters not caught by our initial curation process.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
26

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We are grateful for the work done by the Civit Al community to train and develop 100K+ adapters (LoRAs).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: Both Stylus and StylusDocs are well documented on Github in order to easily reproduce our results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not do any crowdsourcing or human subject studies in this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
27

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not introduce any risks to study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "AMMrlivnD5", "reviewer": "Reviewer_QQJ7", "summary": "This paper works on post-training optimization for image generation with stable diffusion models. They proposed three stages, refiner, retriever and composer, to personalize a SD model for the prompt and thus to generate the perfect images. The experimental result indicate the potential of the proposed method.", "questions": "1. Does the description (or the optimized version) can well represent the adapter?\n2. in Eq.2, why do you directly use betha=0.8? If the 2nd term's value is much bigger than W_base, how do you deal with it?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation makes sense to me and the idea is interesting. Previously, we usually explore prompt engineering to generate a good image, but this paper investigates the adapters and to finalize a suitable model to generate  good image given a fixed prompt. \n2. The post-training optimization consists of three stages, Refiner, Retriever and Composer, the design of the entire method is reasonable.\n3. The experimental results demonstrate the proposed method is promising.", "weaknesses": "1. Efficiency. In the paper, the authors compares the Stylus with the typical SD checkpoint, then the efficiency of stylus is not comparable to SD in terms of memory, CPU, GPU resources.\n2. Fairness. The paper claim advantages over typical SD in terms of diversity and quality. While it is not quite fair. The Stylus has a model personalization process (from the post-training optimization pipelines), while the SD is using a static model checkpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications and experiments address your questions.\n \n**Efficiency.**  We note that Stylus's efficiency for image generation in terms of CPU and GPU resources is near identical to the base Stable Diffusion (SD) model. Aside from merging adapters (e.g. LoRAs) into the base model, which is small, the inference computation remains the same. Additional CPU memory is required to store unmerged adapters, as evidenced by efficient LoRA serving systems such as SLoRA [1] and dLoRA [2].\n\nWe emphasize that users today rely on manual search to identify a helpful subset of adapters. Stylus automates the process of determining the best set of adapters for a given user prompt, improving user efficiency. Furthermore, with recent releases of fast, high quality LLMs (i.e. GPT-4o-mini, Gemini 1.5 Flash), the composer’s latency will continue to reduce over time. We profiled GPT-4o-mini as the composer, which is over 3x faster than Gemini 1.5. This is a small overhead when compared to manual search over adapters.\n\n**Fairness.** Stylus serves as an automatic enhancement to SD, leveraging additional training and data represented by LoRAs. Our retriever ablation (Sec. 4.3.1, Tab. 1) shows that naively selecting adapters (i.e. RAG) can lead to worse performance than the typical SD checkpoint. As such, we include base SD’s CLIP and FID scores as a reference for comparing different approaches to selecting and composing adapters. \n\n**Adapter descriptions.** First, we note that, without access to the original fine-tuning dataset, we cannot guarantee the descriptions are completely error-free. However, we manually inspect commonly selected adapters in StylusDocs, blacklisting adapters that produce low quality images or cases where the StylusDocs description is inconsistent with observed adapter behavior (Sec A.4).  Furthermore, we observe that over 80% of adapters on model platforms (Civit.ai/Huggingface) lack sufficient descriptions. As such, Stylus’s refiner takes in example images from Civit.ai’s model card as a grounding mechanism to generate more detailed and accurate descriptions (Sec A.1).\n\nFurthermore, we add an additional refiner ablation that showcases that better adapter descriptions lead to performance gains. In the following table of CLIP/FID scores, we illustrate Stylus without and with refiner.\n\n| Baselines (CFG=6)        | CLIP: ↑ is better     | FID: ↓ is better      |\n|--------------------------|-----------------------|-----------------------|\n| SD v1.5                  | 27.22                 | 23.96                 |\n| No-Refiner               | 24.91 (-2.31)         | 24.26 (+0.3)          |\n| Gemini-Ultra Refiner     | 27.25 (+0.03)         | 22.05 (-1.91)         |\n| **GPT-4o Refiner**                  | **28.04 (+0.82)**     | **21.96 (-2.00)**     |\n\nHere, the baselines are:\n- SD v1.5 - The base Stable Diffusion model with the RealisticVision checkpoint.\n- No-Refiner: Use base author-provided descriptions from Civit.ai or Huggingface.\n- Gemini-Ultra Refiner: Use Gemini-Ultra as the Refiner’s VLM to generate better adapter descriptions. This is the version of Stylus presented throughout our paper.\n- GPT-4o Refiner: Use GPT-4o as the Refiner’s VLM.\n\nThe quality of author-provided descriptions are poor, leading to worse performance than the typical SD checkpoint. Further improved refiner descriptions from GPT-4o can significantly boost Stylus’s performance, achieving the best textual alignment (CLIP) and image quality (FID) across all baselines, surpassing our original Stylus (with Gemini-Ultra).\n\n**Merging Adapters.** We clarify Eqn. 2 *averages* adapter weights per task and *sums* adapter weights across tasks. We take several measures below to ensure that the second term in Eqn. 2 does not grow too large:\n- Our masking scheme reduces the number of adapters in the final composition of LoRAs. (Sec 3.4)\n- Empirically, with the COCO dataset, we observed the composer identifies at most seven tasks with associated adapters. We also have the option in the composer’s prompt to limit the number of tasks (Tab. 2).\n- β scales down the rate at which the second term grows. We determined β=0.8 prevents highly-weighted adapters from overriding other concepts specified in the prompt, a challenge discussed in Fig. 13(b).\n\nWe appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!\n\n[1] Sheng, Ying, et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arXiv:2311.03285, arXiv, 5 June 2024. arXiv.org, https://doi.org/10.48550/arXiv.2311.03285.\n\n[2] Wu, Bingyang, et al. \"dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving.\" 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), USENIX Association, July 2024, pp. 911-927, Santa Clara, CA, www.usenix.org/conference/osdi24/presentation/wu-bingyang.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTfJNnXBvk", "reviewer": "Reviewer_rCTt", "summary": "The paper proposes Stylus, an approach for automatically selecting and combining fine-tuned adapters on particular tasks to improve the quality of image generation given a prompt. To evaluate Stylus, the paper introduces StylusDocs, a curated dataset containing 75K adapters with pre-computed adapter embeddings. Both the qualitative and quantitative results show that the proposed method outperforms Stable Diffusion and other retrieval methods.", "questions": "No questions beyond the one in the weaknesses.", "limitations": "The paper does not have any obvious limitations that were not discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper explores an interesting topic of automatically selecting and combining fine-tuned adapters on particular tasks to improve the quality of image generation.\n- Both the qualitative and quantitative results are proposing. The proposed methods method improves over other method for both human evaluation and automatic benchmarks.", "weaknesses": "- Some details about the proposed method are missing, making it hard to reproduce. In particular, sections 3.3 and 3.4 about the composer and the masking are not very clear. How are the adapters selected? How is masking applied?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful and enthusiastic review!\n\n**Reproducibility.** We plan to release Stylus and StylusDocs as open-source resources to ensure transparency and reproducibility.\n\n**Adapter selection and masking.** The composer decomposes the prompt into tasks and maps highly relevant adapters to each task. For more details on the Chain-of-Thought prompt [1] used for Stylus’s composer, refer to Table 2. \n\nNext, a subset of candidate adapters are selected via masking for image generation. For each task, a mask either selects A) just one of the task’s adapters, B) all of the task’s adapters, C) or none of the adapters. To get the final selection of adapters, we randomly sample a mask for each task and merge the identified adapters with the original base model.\n\nRegarding the merging step, each adapter weights are first multiplied by the refiner’s recommended adapter weight, α_j (Eqn. 2). For example, [Food elegant style LoRA]( https://civitai.com/models/127450?modelVersionId=139441) recommends α=0.7 weight. Finally, to merge adapters into the base model, adapters weights are *averaged* per task and then *summed* across tasks.\n\nWe appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!\n\n[1] Wei, Jason, et al. “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.” arXiv.Org, 28 Jan. 2022, https://arxiv.org/abs/2201.11903v6.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aPHlY6dDzg", "reviewer": "Reviewer_N5AD", "summary": "The paper addresses the challenge of selecting and composing relevant adapters for generating high-fidelity images with diffusion models. Stylus introduces a three-stage process: refining adapter descriptions, retrieving relevant adapters based on user prompts, and composing them to create the final image. The paper highlights the development of StylusDocs, a dataset featuring 75K adapters with pre-computed embeddings. Evaluation results show that Stylus outperforms baseline models, achieving higher visual fidelity, textual alignment, and image diversity. The system is efficient and suitable for various image-to-image tasks, including translation and inpainting, demonstrating its versatility and effectiveness in improving image generation.", "questions": "None.", "limitations": "None. Great work, Authors!", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- This is a great paper. Original, high quality, clear, and significant. \n- The use of adapters and PEFT will/should continue to increase in the future. The authors present a scalable method to improve text-to-image generation.", "weaknesses": "None. While adapters have been used in the past to improve image generation, this paper provides a much more coherent strategy to integrate them into VLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful and enthusiastic review!  \n\nWe agree the incorporation of adapters/PEFTs [1] will continue to increase and that automatic adapter selection will be critical for managing and navigating the growing ecosystem of fine-tuned models. Stylus demonstrates improved diversity and visual quality evaluated quantitatively with automated metrics (CLIP/FID scores) as well as qualitatively via human evaluation and VLM as a judge.\n\nStylus provides a coherent strategy for composing and routing adapters for VLMs. Our retriever ablation (Sec. 4.3.1, Tab. 1) shows that naively selecting adapters (i.e. RAG) can lead to worse performance than the base SD checkpoint. Strategically composing and routing adapters unlocks a dimension of model performance previously underexplored. \n\nWe are excited to see LLMs used for dynamically selecting among models in related domains, including but not limited to:\n- Automatic construction of agentic workflows/graphs. This includes using Stylus to decompose the task/prompt into a graph of subtasks and identifying which agent, among an ecosystem of agents, is best suited for each subtask.\n- Routing between different base models from different providers to optimize the cost-performance tradeoff.\n- Given a user prompt that requires composing multiple tools/functions, Stylus can identify, retrieve, and then compose the right sets of tools and functions for the LLM to invoke.\n- Domain-specific fine-tuning is an emerging approach to reduce hallucination [2]. Stylus can select the right domain fine-tuned model to maximize factuality.\n\nWe are open to further discussions to improve the paper. Thank you once again for your valuable insights!\n\n[1] Hugging Face Team. \"Parameter-Efficient Fine-Tuning (PEFT) with Hugging Face.\" GitHub, 2023, https://github.com/huggingface/peft.\n\n[2] Tian, Katherine, et al. \"Fine-tuning Language Models for Factuality.\" arXiv, 2023, arxiv.org/abs/2311.08401.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RoCLcl7maE", "reviewer": "Reviewer_wGTi", "summary": "This work presents a novel system, Stylus, designed to enhance the efficiency and effectiveness of generating high-quality images using diffusion models like Stable Diffusion. The key challenge addressed by this work is the automatic selection and composition of relevant fine-tuned adapters from a vast pool of over 100,000 available adapters, which are often poorly documented and highly customized. This work advances the field by providing a robust and automated solution by leveraging the vast number of available adapters.", "questions": "- Motivation of the selected MLLM\n- Additional ablation study\n- Check the quality of adapter descriptions\n- Details on masking process", "limitations": "Maybe potential bias and fairness issues will related with this work", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The creation of StylusDocs, a curated dataset with 75K adapters and pre-computed embeddings, adds substantial value to provide a rich resource for further experimentation and development.\n- Comprehensive Evaluation: The paper provides a thorough evaluation of Stylus across multiple metrics (CLIP, FID) and diverse datasets (Microsoft COCO, PartiPrompts). This robust evaluation framework enhances the credibility of the claimed performance improvements.\n- Open Source and Reproducibility: By planning to release Stylus and StylusDocs as open-source resources, the authors contribute to the transparency and reproducibility of their research. This aligns well with the community’s values and encourages further developments based on their work.", "weaknesses": "- Unclear Motivation for the Method: In the refiner step, the paper does not clearly explain why Gemini Ultra is trusted to generate better adapter descriptions. If other multimodal language models (MLLMs) were used, how would the results differ?\n- Incomplete Ablation Study: The ablation study is not comprehensive as it does not include an ablation of the refiner component. Understanding the impact of the refiner step on the overall performance of Stylus is crucial. \n- Quality Assurance of Adapter Descriptions: The paper does not provide sufficient details on how the quality of the adapter descriptions generated by the refiner is ensured. It is unclear whether any validation or verification steps were taken to confirm the accuracy and reliability of these descriptions. \n- Insufficient Description of Masking Process: The description of the masking process is not detailed enough. Specifically, the meaning of α_j in Equation 2 and the function Mask() are not adequately explained. Additionally, the masking process is not reflected in Figure 2, which outlines the Stylus algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the thoughtful review and the constructive feedback! We hope the following clarifications address all the questions raised.\n\n**Quality Assurance for Adapters.** We have taken careful preemptive measures to filter out problematic adapters and have outlined in Appendix A.7 a continual curation process, where we invite community members to report any adapters missed in our initial curation. Below, we highlight some key preventative measures for quality assurance and refer readers to Appendix A.7 and A.8 for a full discussion on safety and reliability of Stylus.\n\n- **Safety.** Although LLM safety remains an actively evolving research area [1], StylusDocs employs a multi-stage filtering pipeline to identify and exclude problematic adapters. First, all explicit adapters tagged by Civit.ai are excluded from StylusDocs (Sec 3.2). Second, we use filters from Google’s VertexAI API to reject unsafe adapters based on LLM-generated descriptions. This especially catches problematic adapters that have innocuous original descriptions (Sec. A.7).\n- **Accuracy/Reliability.** We note that, without access to the original fine-tuning dataset, we cannot guarantee the descriptions are completely error-free. However, we manually inspect commonly selected adapters in StylusDocs, blacklisting adapters that produce low quality images or cases where the StylusDocs description is inconsistent with observed adapter behavior (Sec A.4). Furthermore, authors are incentivized to oversell the model’s abilities. As such, Stylus’s refiner takes in example images in Civit.ai’s model card as a grounding mechanism to generate more accurate descriptions (Sec A.1).\n\nEven with the safety measures we’ve taken, we acknowledge Stylus has the same risks of misuse as other public domain image generation tools including improper use for misinformation, producing explicit content, and reproducing copy-righted material from the training data. We emphasize the research prototype is not meant to be used in production without further application informed guardrails.\n\n**Refiner Ablation.** In the following table containing CLIP and FID scores, we ablate Stylus's refiner:\n\n| Baselines (CFG=6)        | CLIP: ↑ is better     | FID: ↓ is better      |\n|--------------------------|-----------------------|-----------------------|\n| SD v1.5                  | 27.22                 | 23.96                 |\n| No-Refiner               | 24.91 (-2.31)         | 24.26 (+0.3)          |\n| Gemini-Ultra Refiner     | 27.25 (+0.03)         | 22.05 (-1.91)         |\n| **GPT-4o Refiner**                  | **28.04 (+0.82)**     | **21.96 (-2.00)**     |\n\nThe baselines are:\n- SD v1.5 - The base Stable Diffusion model with the RealisticVision checkpoint.\n- No-Refiner: Use base author-provided descriptions from Civit.ai or Huggingface.\n- Gemini-Ultra Refiner: Use Gemini-Ultra as the Refiner’s VLM to generate better adapter descriptions. This is the version of Stylus presented throughout our paper.\n- GPT-4o Refiner: Use GPT-4o as the Refiner’s VLM to generate better adapter descriptions.\n\nThese results show that a refiner is indeed important for textual alignment with the prompt. In fact, without a refiner VLM, Stylus performs poorly and chooses adapters that the composer thinks are aligned with the prompt but are not in practice. As a result, Stylus chooses adapters that hurt textual alignment (CLIP) and image quality (FID) and performs worse than base SD.\n\nFurthermore, GPT-4o baseline performs much better than Gemini-Ultra, showing that better refiner descriptions can better aid the composer in selecting the right adapters. This also suggests that the Stylus’s performance is independent of Gemini-specific capabilities and benefits from further improved visual-language reasoning capabilities.\n\n**Choice of Gemini for Refiner.** We chose the Gemini class of models since it has mature safety guardrailing. Specifically, Google’s VertexAI API provides stringent safety settings to block explicit content for the input prompt. Safety filters helped us filter out around 30% of original adapters that were tagged as non-explicit by Civit.ai.\n\n**Masking.** The composer decomposes a prompt into tasks and assigns highly-aligned adapters per task. Next, a subset of candidate adapters are selected via masking for image generation. For each task, a mask either selects A) just one of the task’s adapters, B) all of the task’s adapters, C) or none of the adapters. To get the final selection of adapters, we randomly sample a mask for each task and merge the identified adapters to the original base model.\n\nRegarding merging adapters, each adapter weights are first multiplied by the refiner’s recommended adapter weight, α_j (Eqn. 2). For example, [Food Elegant Style LoRA](https://civitai.com/models/127450?modelVersionId=139441) recommends α=0.7 weight. Finally, to merge adapters into the base model, adapters weights are *averaged* per task and then *summed* across tasks.\n\nWe appreciate your feedback and believe these clarifications should address your concerns. We are open to further discussions to improve the paper. Thank you once again for your valuable insights!\n\n[1] Hendrycks, Dan, et al. \"Unsolved Problems in ML Safety.\" arXiv, 29 Sep. 2021, arXiv:2109.13916.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all the reviewers for their insightful feedback and enthusiastic reviews! To name just a few comments, reviewers acknowledged that Stylus is novel,

> This is a great paper. Original, high quality, clear, and significant. (Reviewer N5AD)

> This work presents a novel system, Stylus  (Reviewer wGTi)

> The idea is interesting … this paper investigates the adapters and to finalize a suitable model to generate good image given a fixed prompt. (Reviewer QQJ7)

timely and impactful,

> advances the field by providing a robust and automated solution (Reviewer wGTi)

> The use of adapters and PEFT will/should continue to increase in the future. (Reviewer N5AD)

especially for our open-source StylusDocs dataset,

> The creation of StylusDocs, a curated dataset with 75K adapters and pre-computed embeddings, adds substantial value to provide a rich resource for further experimentation and development.  (Reviewer wGTi)

and is comprehensively evaluated.

>Comprehensive Evaluation: The paper provides a thorough evaluation of Stylus across multiple metrics (CLIP, FID) and diverse datasets (Microsoft COCO, PartiPrompts). This robust evaluation framework enhances the credibility of the claimed performance improvements. (Reviewer wGTi)


___
___


We’d also like to highlight some of the shared feedback across reviewer rebuttals.

**Impact of Refiner (QQJ7, wGTi).** Our additional ablation experiments demonstrate that Stylus’s performance benefits significantly from the high quality of adapter descriptions provided by the VLM-based Refiner.

| Baselines (CFG=6)        | CLIP: ↑ is better     | FID: ↓ is better      |
|--------------------------|-----------------------|-----------------------|
| SD v1.5                  | 27.22                 | 23.96                 |
| No-Refiner               | 24.91 (-2.31)         | 24.26 (+0.3)          |
| Gemini-Ultra Refiner     | 27.25 (+0.03)         | 22.05 (-1.91)         |
| **GPT-4o Refiner**                  | **28.04 (+0.82)**     | **21.96 (-2.00)**     |

Here, the baselines are:
- SD v1.5: The base Stable Diffusion model with the RealisticVision checkpoint
- No-Refiner: Use base author-provided descriptions from Civit.ai or Huggingface.
- Gemini-Ultra Refiner: Use Gemini-Ultra as the Refiner’s VLM to generate better adapter descriptions. This is the version of Stylus presented throughout our paper.
- GPT-4o Refiner: Use GPT-4o as the Refiner’s VLM.
 
Without the refiner, the poor quality of author-provided descriptions results in Stylus performing worse than SDv1.5. However, the high quality of adapter descriptions from the GPT-4o Refiner results in the best performance, surpassing Gemini-Ultra Refiner, the original refiner VLM.

**Safety and Reliability of Adapter Descriptions (QQJ7, wGTi).** Stylus ensures adapter *safety* through a multi-stage filtering pipeline, initially excluding all explicitly tagged adapters by Civit.ai (Sec 3.2), followed by using Google's VertexAI API filters to reject unsafe adapters based on LLM-generated descriptions (Sec. A.7). For *reliability/accuracy,* Stylus's refiner uses example images from Civit.ai’s model card as a grounding mechanism to generate more accurate descriptions (Sec A.1), and we manually inspect and blacklist low-quality, explicit, or highly-inaccurate adapters (Sec A.4).

**Masking and Merging Clarification (QQJ7, rCTt, wGTi).** Reviewers asked for more clarity on Stylus’s masking and merging steps. Recall that the composer decomposes the prompt into tasks and maps highly relevant adapters to each task. For each task, a mask either selects A) just one of the task’s adapters, B) all of the task’s adapters, C) or none of the adapters. The selected adapters are then merged, with adapter weights *averaged* per task and then *summed* across tasks.

Meta-review: This paper explores systematically using adapters to improve the quality of diffusion model generated images, at reduced cost and increased efficiency. The paper studies how to automatically select adapters from a large library based on the given prompt.

The reviewers were all enthusiastic about the paper, and it would have a broad appeal to the NeurIPS audience. I therefore recommend an oral.
Decision: Accept (oral)