Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild

Authors: jianan  yang
chenchao  gao
zhiqing  xiao
junbo  zhao
sai  wu
gang  chen
haobo  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 11:33:44

Abstract: The recent large-scale text-to-image generative models have attained unprecedented performance, while people established *adaptor* modules like LoRA and DreamBooth to extend this performance to even more unseen concept tokens. However, we empirically find that this workflow often fails to accurately depict the *out-of-distribution* concepts. This failure is highly related to the low quality of training data. To resolve this, we present a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD). Our framework follows the active learning paradigm which includes high-quality data accumulation and adaptor training, enabling a finer-grained enhancement of generative results. The *aesthetics* score and *concept-matching* score are two major factors that impact the quality of synthetic results. One key component of CATOD is the weighted scoring system that automatically balances between these two scores and we also offer comprehensive theoretical analysis for this point. Then, it determines how to select data and schedule the adaptor training based on this scoring system. The extensive results show that CATOD significantly outperforms the prior approaches with an 11.10 boost on the CLIP score and a 33.08% decrease on the CMMD metric.

Main paper: 
 introduction: 
 The generative modeling for text-to-image has attained unprecedented performance most recently [37, 45, 41, 49] . Notably, by training over billions of text-image data pairs [52, 51] , the family of diffusion models has allowed high-fidelity image synthesis directed by the prompt provided in production. Despite their massive successes, these models still fail to generate images with decent quality and matching semantics, when encountering prompts that contain unseen or out-of-distribution concept tokens [60, 12, 21] . Simply put, the reason causing this failure is due to that the training set is not unbounded with limited variations. On the side of production, this limitation could significantly impact the practicality of this technique in real-world applications.To deal with such concepts, recent works have resorted to adaptors such as Textual Inversion [18] , DreamBooth [47, 58, 48] , and LoRA [25, 66, 76] , which tunes only a small part of the text-to-image model or insert extra modules. These adaptors largely reduce the training costs, and more importantly, preserve the visual aesthetic information originally learned by the underlying model. One observation is that concepts with higher CMMD scores are notably more challenging for the underlying model to generate (the second row). Additionally, we also notice that a higher CMMD value leads to a more notable loss of visual details when training adaptors (the third row).However, in this paper, we have found that recent works still struggle to accurately depict the visual details of out-of-distribution concepts (with a CMMD score above 3.5), as illustrated in Figure 1 . Adapters like LoRA are able to accurately represent the shape and color of OOD concepts compared to the generative results before adaptor training, but they fall short when it comes to finer details such as texture, contours, and patterns. This issue arises from the fact that current studies predominantly focus on variations of in-distribution (ID) concepts (e.g., humans, dogs, and cats) while ignoring the out-of-distribution (OOD) ones. This failure motivates us to think about what makes the problem of distorted visual details happen when training adaptors.In Figure 2 , we observed that how an adaptor depicts OOD concepts can be significantly influenced by the quality of the training data. If the model is trained on samples containing disruptive objects, the resulting generative outputs are likely to reflect these disruptive elements. When the training data contains images with vague or very small instances of the OOD concept, the generative results may appear low-quality. In contrast, the high-quality data for adapting OOD concepts usually contains a single and clear object corresponding to the given concept, which is highly distinguishable from the background and other types of objects and helps produce accurate results with high-fidelity. However, manually picking such high-quality data requires much human labor and expertise, which may crucially limit the versatility of text-to-image models. Therefore, an effective paradigm to locate high-quality samples of OOD concepts is important for the practical shipping of this field.To this end, we developed a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (referred to as CATOD) which aims to identify high-quality samples to guide the adaptor training. This framework follows the Active Learning (AL) paradigm [54, 44] , involving iteratively accumulating training data and updating the adaptor. The profound motivation of this approach is to comprehensively model the interaction between training data and the underlying text-to-image model. Specifically, CATOD includes two interconnected scores: the aesthetic score and the conceptmatching score, following the observation that object clarity and uniqueness largely impact generative results, as illustrated in Figure 2 . Based on this, we devised a weighted scoring system that adapts itself according to the adaptor to select high-quality data while also properly balancing the two scores. With the carefully selected high-quality data, we schedule the adaptor training based on the quality of generative results evaluated through this scoring system. In summary, our contributions are as follows: (i)-We have identified the challenge of adapting text-to-image models to out-of-distribution (OOD) concepts, where recent studies often struggle to accurately depict them; (ii)-We have introduced a framework called CATOD that iteratively updates training data and the adaptor to generate OOD concepts precisely; (iii)-Our extensive experiments verified that CATOD achieves significant performance gain with up to 11.10 on the CLIP score and 33.08% on the CMMD metric; (iv)-We have also offered theoretical insights into the key factors: aesthetics and concept-matching, which contribute to the effectiveness of our method. When an object within the image is too small, the results may not accurately represent the intended concepts (Middle). In contrast, if the image contains a high-fidelity object without disruptive elements (Right), the model is more likely to generate the desired result accurately.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes]Justification: We have clearly summarized our contribution in the abstract and in the last paragraph of the introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
ood problem in latent diffusion models: 
 Revisiting Latent Diffusion Models (LDMs). Latent Diffusion Models (LDMs) [45] comprise two components: a diffusion process operating the latent space and an auto-encoder which contains an encoder E mapping an image into the latent space and a decoder D that reconstruct images from latent codes. Furthermore, the diffusion process can be conditioned on the output of text embedding models, enabling the auto-encoder to integrate the information derived from texts. Let x be the image, the CLIP textual encoder c θ that maps the corresponding text y into the latent space, the LDM loss is:L LDM (x, y) := E z∼E(x),ϵ∼N (0,1),t ∥ϵ -ϵ θ (z t , t, c θ (y))EQUATIONwhere t denotes the time step, z t denotes the latent code noised at time step t, when ϵ, ϵ θ represents the noised samples and the denoising U-Net [46] , respectively. Through this noising-denoising procedure applied to the latent codes, LDM enables the underlying model to integrate information derived from texts into the visual results, while also allowing more flexibility to produce images.The OOD Concepts for LDMs. Intuitively, out-of-distribution (OOD) concepts refer to the category of data whose distribution deviates significantly from what the model has learned. This degree of drifting can be quantified by an MMD score [27] , which evaluates the discrepancy between ground-truth images and the generative results in the image latent space. Formally, for two probability distributions P and Q, the MMD distance with respect to a positive definite kernel k is:EQUATIONwhere x p , x ′ p independently follow the distribution P while x q , x ′ q independently follow the distribution Q, with k the Gaussian RBF kernel [17] . In our implementation, we sample two sets of vectors from the distribution P and Q, then use CLIP embeddings [40] to calculate this score, which is also named as CMMD [27] . We set a concept with a high CMMD score (above 3.5) as an OOD concept.OOD Concepts are Hard to Adapt. Recent text-to-image LDMs [45, 39] have achieved unprecedented performance on a wide range of concept tokens. However, we have found that there still exist many concepts that make LDMs fail after the adapter is fully trained. As we show in Figure 1 , there are several discoveries: (i)-The concepts with higher CMMD scores are much more challenging for the underlying model to generate or adapt. The concepts with a CMMD score above 3.0 show explicit wrong visual details. For Axolotl and Frilled Lizards with CMMD above 4.0, the LDMs even generate the wrong species; (ii)-A higher CMMD score indicates a more severe loss of visual details when training adaptors. The concepts with CMMD scores above 3.5 in Axolotls and Emperor Penguin Chicks, show explicit distorted visual details, like color, texture, and delicate details. For Axolotl, the generative results show a wrong number and wrong positions of amateurs. For Emperor Penguin Chicks, the generative results show the wrong fur color of their heads and wings. To summarize, the higher the CMMD score of a concept, the more difficult it is for LDMs to adapt.The High-Quality Matters. We further observe that the generative results are quite sensitive to training data when training adaptors meet OOD concepts. As shown in Figure 2 , when the training images contain disruptive elements, the visual features of these disruptive elements will be easily introduced into the generative results. For example, when an adult emperor penguin appears in training data, then the black fur on their back can easily appear when generating their chicks, despite that their chicks have white fluffs as shown in the left part of Figure 2 . If the object of the desired concept appears to be too small within the image, then the generative results tend to be bad-looking, since the necessary visual details are not fully identified. In the middle part of Figure 2 , we can see that distorted visual details like texture and shape in axolotl and emperor penguin chicks largely harm the aesthetics of the generated image. To generate images correctly and good-looking, we require enough amount of images with objects of high fidelity and do not contain disruptive elements, namely High-Quality samples as shown in the right part of Figure 2 . Therefore, we aim to devise an effective data selection strategy for locating those high-quality images for these OOD concepts.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: We believe that the limitation should not be regarded as an integral part of this paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overall architecture: 
 We aim to adapt OOD concepts to LDMs correctly by an iterative data selection criterion that locates high-quality data and training adaptors accordingly as shown in Figure 3 . Consequently, our method would alleviate issues introduced by disruptive elements, e.g., irrelevant objects, and blurry images, while maintaining high fidelity of generative results. The core to CATOD is a scoring system, which consists of an aesthetic scorer and a concept-matching scorer, aiming to resolve the problems of incorrectly introduced visual details and distorted objects we have observed in Section 2. By properly trading off between these two scores, CATOD locates the most valuable samples for adapting underlying LDMs to OOD concepts and making the desired OOD concept depicted correctly.As described above, we need to estimate the potential impact of real-world samples over LDMs, according to which we select the most high-quality samples for training. In Section 2, we have observed two major factors that significantly impact generated image quality: object clarity and disruptive elements. Diving into the loss term L LDM (x, y) in Eq. ( 1), we may also observe that the training set X T should be optimized towards both the underlying model (including the embedded adaptor) and the conditional text y, which indicates two important factors: object clarity and conceptmatching achieved by preserving aesthetic information originally learned by LDMs and accurate image-text matching, respectively. Therefore, we attribute the quality of images to aesthetic and concept-matching and design two decoupled scorers accordingly.The Aesthetic Scorer. Aesthetic evaluation is a long-standing field, which comprehensively considers whether the lighting, contrast, texture, and other photographic factors of an image are consistent with human aesthetics. The general aesthetic scorer can be simply described as p = S aes (x), where S aes indicates the aesthetic scoring model, p denotes the predicted score, and x represents the input image. Following the work of PA-IAA [33] , we fine-tune the generic aesthetic model and make it adapt to OOD concepts with personalized preference scores. In general, we assign a high score to the samples within the training set X T , assign a low score to samples from irrelevant categories and samples generated by underlying LDMs without an adaptor, and use them to fine-tune the aesthetic scorer. More details of this personalization are given in the Appendix B.3.The Concept-Matching Scorer. Intuitively, concept-matching describes whether the OOD concepts get perfectly reflected in the generated results. A similar task is image retrieval, which is designed to retrieve images containing objects describing the desired concepts. Since image retrieval also relies on feature maps and some sort of matching score to retrieve images, we adopt matching score from VLAD-related works [28, 64, 55] as follows:EQUATIONwhere ϕ(r x , r k ) = 1 if r k is the closet representation for r x and is set to 0 otherwise. Simply put, VLAD regards the extracted representations for X T as a codebook and maps each image to its nearest code. Note that samples within X T mostly consist of clean and clear samples, this score is sufficient to quantify whether the object if exists in the given image distinguishes itself from other photographic components and matches the given concept.Optimization with Active Learning (AL). Aiming to mitigate the problems caused by low-quality training samples, we proactively integrate the training data X T into our objective as follows:EQUATIONSince the optimal set is initially unknown, a one-step optimization can easily lead to convergence to local optima. Therefore, we use an iterative paradigm to optimize adaptor A and training data X T , respectively. In our implementation, we adopt the paradigm of AL [44, 54] to perform the optimization ofX (t)T by data accumulation before training adaptors, with t denotes the time step:EQUATION)where b is the number of samples added to the training pool at each cycle. The main reason for using AL is its preferred sample efficiency, with better controllable data bias management [15, 50] . Instead of repeatedly selecting data from the whole real-world data pool, AL provides a more efficient procedure to optimize training data by using data selection to accumulate high-quality training data. To this stage, the learning procedure of CATOD is relatively clear: the sample pool is progressively accumulated -by Eq. ( 5) -and the optimization of the adaptor A is straightforward (Fig. 3 ).Remember that AL involves iteratively updating the adaptor and the training data, it is important to design a training schedule for adaptors and determine how to acquire high quality based on the two scorers mentioned above. The primary objective of this design is to achieve a dynamic trade-off between the two scores. The specific details of these designs are described below.The Active Schedule for Training Adaptors. The training schedule has been found crucial for successful adaption [47, 74, 65] . Since our training data continuously expands as the learning cycle of AL proceeds, the training schedule will be even more important. To arrange this schedule, we first calculate the aesthetics score γ aes (A) =1 |g A (X T )| xg∈g A (X T ) S aes (x g ) and concept-matching score γ con (A) = 1 |g A (X T )|xg∈g A (X T ) S con (x g ) of the adaptor A based on its generative results g A (X T ), in which both the aesthetic score γ aes (A) and γ con (A) range from 0 to 10. Then, we use a trigonometric indicator that comprehensively measures its performance:EQUATIONNotably, γ(A) peaks when the two scores get close or around the common value of 5.0 for most samples. Meanwhile, it bottoms when the scoring exhibits a stronger signal of being biased (to either side). The properties emphasize that adapters balancing the two factors are of better quality. In training CATOD, we use γ(A) as a signal to reduce the learning rate and stop training in time.Trading-off the Two Scores in Data Acquisition. After getting the adaptor A, we continue to select the most suitable samples for the next-cycle training. Notice that whether newly selected images enhance the adaptor depends on both aesthetics and concept-matching, we ought to adjust our preferences according to the adaptor. In more detail, when the adaptor has high aesthetics but does not accurately depict the OOD concept, samples with high concept-matching scores better enhance the adaptor; when the adaptor fails to exhibit photographic attributes consistent with humans, samples with high aesthetics are preferred. To implement this preference, our acquisition score is:EQUATIONThis formulation offers some meritable advantages. On one hand, the balancing terms are not prefixed or manually tuned, but dynamically dependent on the scores marginalized over the current generations. Further, when the score of either side gets larger, the corresponding balancing coefficient, in turn, decreases, thus attaining a proper trade-off. As a result, this mechanism encourages an alternation of sample selection towards both scoring metrics by selecting Top-K samples to add to the training data X T accordingly, which effectively guarantees the sample set diversity.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: This paper does not provide the dataset regarding the risk of anonymity, despite that the dataset is just a collection of publicly available datasets.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
theoretical insights: 
 This section presents our theoretical analyses of why aesthetic/concept-matching scores work in OOD Adaption. Specifically, we derive the distance between real-world data distributions and synthetic data distributions and then induce the important factors that affect this distance. We first introduce the minimum mean square error (MMSE) [22, 11, 6] to measure the discrepancy between distributions: Definition 4.1. The minimum mean square error (MMSE) of estimating an input random vector X ∈ R n from an observation/output X ∈ R k is defined asEQUATIONin which M(R n ) denotes the space consisting of all measurable functions on R n .Notice that we are trying to produce the best generation results which are initially unknown, our adaption task can be also regarded as estimating the optimal distribution with carefully selected data. By denoting the ideal generative results and the ideal adaptor with random variables X G and A * , respectively, we conclude that the LDM loss L LDM is consistent with this MMSE term: Theorem 4.2. Let L LDM be the LDM loss following Eq. 1, and the image space lies within R n . Then there always exists an ideal random vector X G ∈ R n and an adaptor A * for LDM, such that arg minEQUATIONBased on the preceding deduction, we have confirmed that the change in MMSE can also indicate the change in LDMs. To dive deep into the MMSE term, we further decompose it as follows: Theorem 4.3. (Pythagorean Theorem for MMSE [13] .) Following Theorem 4.2, by setting f to the generative model g A , the MMSE term in Eq. ( 8) can be decomposed into two terms as follows:EQUATIONin which A * denotes a potential ideal adaptor containing information for text y. Notice that the generative results X G relies on the training set X T , g A (X T ) can also be written as X G |X T .Upon revisiting the two terms, we can gain a deeper understanding of the relationship between MMSE and aesthetic/concept-matching score: (i)-The first term is focused on estimating the difference between the generative distribution X G |X T and the ideal distribution of X G . This assessment helps  
experiments: 
 In this section, we present the main experimental results both qualitatively and quantitatively. To evaluate our proposed CATOD, we combine it with several works for adaption, i.e. DreamBooth [47] ,Textual Inversion [18] , and LoRA [25] . More experimental results can be found in the Appendix.The source code is attached in the Supplementary.Datasets. We test our method on datasets with 25 OOD concepts that can hardly be generated through prompt engineering on the text-to-image model. This dataset consists of 5 categories: insect, lizard, penguin, seafish, and snake, and each category contains data from 5 OOD concepts. Each concept has 1,000 examples in total with 100 samples left out for validation. The dataset is collected from publicly available datasets including ImageNet, iNaturalist 2018 [67] , IP102 [71] .Implementation Details. We conduct the active generation experiments on our proposed CATOD and three representative adaptors, i.e. DreamBooth [47] , Textual Inversion [18] (termed as TI in the paper), and LoRA [25] . Since there are currently no available studies that focus on locating "high-quality" samples for training, we apply random sampling (RAND), and CLIP-score-based sampling (CLIP) for each baseline in our active learning setting. Each experiment starts with 20 randomly sampled instances, and we conducted 5 cycles of data accumulation in which we selected 20 "good" samples to add to the training pool. We train 20 epochs for all combinations of adaption techniques and sampling strategies in each active learning cycle, with a batch size of 1. Furthermore, we generate 100 images for each concept for evaluation. We use the commonly adopted Stable Diffusion 2.0 pre-trained on LAION-5B [51] following Rombach's work [45] .Evaluation Metrics. We evaluate the quality of our generated images with the widely used CLIP score [20] and the recently proposed CMMD score [27] , which quantify the model performance in two aspects. Specifically, the CLIP score measures how generated images match the given text, which is expected to be as high as possible. Meanwhile, the CMMD score evaluates the discrepancy between generated images with the real ones, indicating better generative results with lower values.To evaluate the performance of CATOD on OOD concepts, we test it on all 25 target concepts one by one separately and report the average performance of 5 concepts within each category in Table 1 . We show the superior results of our CATOD with respect to both qualitative and quantitative comparisons.Qualitative Comparisons. We qualitatively compare CATOD with other sampling strategies according to their generated images based on LoRA [25] . With random sampling (RAND), we observe that the generative results only partially learned some photographic attributes like color and texture, but failed to make the objects have the correct appearance and shape. CLIP-based sampling (CLIP) somehow produces the corrected shape of the concept but still fails to capture the necessary details for describing the object. In comparison, our proposed CATOD successfully matches all the photographic attributes, while also guaranteeing the image aesthetics.To further look at how CATOD learns the photographic attributes that precisely match the concept, we show samples generated from different cycles in Fig. 7 . We can see that some attributes like color, and texture are already learned in cycle 1, but the shape of the object does not match the ground-truth ones. From cycle 1 to 3, CATOD shows a clear shape modification, making the objects more like the real ones. From cycles 3 to 5, an iterative refinement on more photographic details like light, contrast, and other minor modifications (like the beak for penguins and antenna for axolotls) is shown in generative results, making them hard to distinguish from the ground-truth ones even with a careful look. At cycle 5 and later cycles, the image quality stabilizes and we can hardly see enhancement apart from image diversity. To conclude, we can see an explicit attribute matching process from easy ones to the finer ones within CATOD, showing the importance and effectiveness of iterative training.Quantitative Comparisons. Table 1 reports CLIP scores [20] and CMMD scores [27] from each strategy with models trained on 25 different OOD concepts and evaluated through the generative results. Specifically, we evaluate the performance of CATOD on each concept and average the results within each category in Table 1 . We have the following conclusions: (1) the average performance of our strategies outperforms all compared methods by a 0.56∼11.10 CLIP score, justifying the effectiveness of locating aesthetic and clean samples over text-image matching. (2) CATOD also brings a 0.12∼0.54 CMMD decrease on various frameworks and concepts, indicating better image alignment with proper sampling guided by CATOD. To summarize, both image-matching and textmatching scores exhibit better results compared to the baselines, suggesting that our proposed CATOD significantly outperforms other strategies, which is consistent with the qualitative results in Fig. 4 . To further investigate whether CATOD could adapt multiple concepts simultaneously, we group the 25 concepts by category and train the adaptation model on each category. To be specific, we compare our baselines using a generated set consisting of 500 images (100 images per concept) and exhibit our results in Table 2 . Following the setting of single-concept adaption, we conduct 10 cycles of data accumulation and get 200 samples at last, since multi-concept adaption requires more data. We still observe a notable performance gain with a 1.56∼5.07 CLIP score increase and a 0.12∼0.86 CMMD score decrease compared to baselines. These results verify that our CATOD still achieves better performance on multi-concept adaption.We verify the efficacy of all components in our proposed CATOD in Table 3 and 4 , including the aesthetical scoring module and the concept-matching mechanism within the weighted scoring system.W/O Aesthetic Score. First, we validate the effectiveness of CATOD by removing the aesthetic scores. A significant decrease in performance (up to 8.55) on the CLIP score can be observed in Table 3 . We attribute this decline to the fact that matching-based metrics prioritize image representations over the given concept, leading to a deterioration in image-text matching.The Type of Aesthetic Scores. To further investigate the impact of aesthetic scores, we have also applied different types of aesthetics with CATOD in Table 4 . We observed that recent state-of-theart aesthetic evaluations did not improve OOD adaption and even led to minor performance loss. This could be attributed to the fact that these models were originally designed for general aesthetic assessments, whereas our aesthetic scorer is highly personalized towards specific OOD concepts.W/O Concept-Matching Score. After removing the concept-matching scorer, we observed that the adaptor tends to perform better compared to just removing the aesthetic scorers. This might be due to the aesthetic scorer being designed based on CLIP backbones, showing some consistency with the CLIP score. However, it still demonstrates limitations based on the image-matching score CMMD. While these aesthetic qualities partly describe the clarity and accuracy of the object, they do not adequately focus on the image representation space.The Weighted Score. We have confirmed the effectiveness of balancing two scores by simply adding them together to guide CATOD (Table 3 , Line 5). We observed that this resulted in consistent performance loss for both the CLIP score (up to 1.65) and the CMMD score (up to 0.22). This suggests that both text-image matching and image-to-image matching are affected by the trade-off between aesthetics and concept matching, further emphasizing the importance of these two scores. 
related work: 
 Personalized Text-to-image Synthesis with Adaptors. The task of text-to-image generation involves creating specific images based on text descriptions [3, 73, 79, 4] , and has achieved impressive performance with state-of-the-art diffusion models [41, 45, 39] . Therefore, adapting large-scale text-to-image models to a specific concept while also preserving this amazing performance, i.e. personalization [7] , has become another recent research interest. But this is often difficult since re-training a model with an expanded dataset for each new concept is prohibitively expensive while fine-tuning the whole model [12, 32] or transformation modules [83, 20, 60] on few examples typically leads to some rate of forgetting [30] . Therefore, adaptors such as Textual Inversion [18] , DreamBooth [47, 48, 5, 58] , LoRA [25, 66, 86, 76] , along with some other works [70, 19] derived from them, have become more commonly adopted. Typically, they focus on a small but crucial part of the model or extra networks inserted into underlying models, thus more computationally-efficient, while also preserving the efficacy of the underlying models with lower computational costs. For example, textual inversion (TI) [18] represents the newly-given concept with pseudo word [42] and remapping it to another carefully trained embedding in the text-encoding space, guided by few images. Despite their computational efficiency, these approaches are still facing difficulties dealing with out-of-distribution concepts as we observe in Section 2.Active Learning and Selection. Active Learning is a machine learning paradigm that involves actively selecting the most suitable data for training models from external data sources [44, 63] . The most crucial part of active learning is the strategy to locate the optimal data batch. Current studies can be roughly categorized as follows: (a) Score-based methods that prefer the samples with the highest information scores [36, 69, 10] ; (b) Representation-based methods searching for the samples that are the most representative of the underlying data distribution [53, 1, 62] . The Active Selection paradigm within Active Learning serves as an efficient and powerful dataset curation tool (i.e. adaption), leading to numerous studies adopting this paradigm from a wide range of subjects [68, 24, 8] . Meanwhile, due to the effectiveness of Active Learning in selecting the most suitable training samples, recent studies have utilized similar sampling strategies to address challenges associated with long-tailed distributions [56, 57] and noisy data [72] . Given that Out-Of-Distribution (OOD) concepts in Latent Diffusion Models (LDMs) often involve unseen or long-tailed concept tokens [79] , this motivates us to leverage Active Learning for selecting samples that are well-suited for training adaptors.In this work, we focus on scored-based strategies in two-fold: aesthetics and concept-matching.Image Aesthetics Assessment (IAA) aims at evaluating image aesthetics computationally and automatically [9] , while automatically assessing image aesthetics is useful for many applications [35, 29, 14] .To take a step further, personalized image aesthetics assessment (PIAA) [43, 85, 75] was proposed to capture unique aesthetic preferences, consistent with our goal to adjust our paradigms accordingly different concepts. At the same time, "Concept-matching" is adopted from the field of image retrieval, in which we search for relevant images in an image gallery by analyzing the visual content (e.g., objects, colors, textures, shapes etc.), given a query image [61, 31] . To design a paradigm that automatically meets the harsh requirements for adaptor training, we consider both two factors.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All experimental details are provided in Appendix B.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
conclusion: 
 We propose CATOD, an enhanced, data-efficient, and practically useful version of the OOD concept adaptation for AIGC. This method is encapsulated in an active-learned paradigm with carefully designed acquisitional scoring mechanisms. CATOD significantly outperforms the prior approaches in many (if not most) aspects including generation quality, concept matches, technological robustness, data efficiency, etc. In the future, we hope to ship CATOD to the open-source community so as to absorb more OOD concepts that were originally uncovered.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our experiments do not have error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). Justification: This has been briefly discussed in our conclusion section. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: This section provides a complete derivation for the analysis given in Section 4. In brief, we first link L LDM to a minimum mean square error (MMSE) term in Theorem A.2 (i.e. Theorem 3.2 within the main context), then decompose the MMSE term to see how we minimize MMSE in different aspects in Theorem A.3 (i.e. Theorem 3.3 within the main context) in the main context. For convenience, we reclaim some of the formulations at the beginning of this section: Definition A.1. The minimum mean square error (MMSE) of estimating an input random vector X ∈ R n from an observation/output X ∈ R k is defined asEQUATIONin which M(R n ) denotes the space consisting of all measurable functions on R n .Based on Eq. ( 1), we can adapt the LDM to an arbitrary concept y with a corresponding image set X that describes this concept. Typical adaptations are mostly based on fine-tuning all the parameters θ, which is quite costly and requires a lot of data. However, recent studies have found that training only a small part of the model [18, 47] or inserting extra networks [25, 26] could attain the same performance, which largely alleviates the computational burden with far fewer samples need for training. We call this part of parameters or networks as "adaptors" and denote them by A. Then, the fine-tuning process of the adaptors, also named adaption, can be formulated as follows:EQUATIONTherefore, v be a conditioning vector corresponding to some given text y, the LDM loss is:EQUATIONFollowing the definition of LDM loss, we continue to link LDM loss to the MMSE term in the theorem as follows (in which we omit the parameter y since it is independent of other parameters): Theorem A.2. Let L LDM be the LDM loss following Eq. ( 13), and the image space lies within R n . Then there always exists an ideal random vector X G ∈ R n and a condition vector v * within the text embedding space for LDM, such that arg minX T ∈R n MMSE( X G |X T ) = arg min X T ∈R n E x∼X T [L LDM (x, A * )] .Proof. By denoting an ideal generative distribution with X G , which is produced by the generative model with a minimized LDM loss, we continue our proof. Set f A (X T ) = E x∼X T [x, A], we can see that f A is also a functional controlled by the adaptor A. Following the definition of MMSE and a fixed X T , we set A * as:EQUATIONWith this A * , and the Universal Approximation Theorem [38] for all deep learning models, f A * becomes the functional needed to quantify the MMSE term following its definition. Therefore, with proper X G , A * , we haveMMSE( X G |X T ) = E x∼X T [L LDM (x, A * )] ,thus completing the proof of this theorem.Theorem A.3. (Pythagorean Theorem for MMSE [13] .) Following Theorem A.2, by setting f to the generative model g A * with an ideal adaptor A * containing sufficient OOD concept information, rewrite g A (X T ) to X G |X T , and the minimum mean square error (MMSE) in Eq. ( 11) can be decomposed into two terms:EQUATIONProof. This equation can be further decomposed into two terms:E X G -E [X G |X T ] ϕ(X T ) = E ( X G -X G )ϕ(X T ) + E [X G -E [X G |X T ] ϕ(X T )] .Since X G -X G represents the variance for generative results and is orthogonal to X T , the first term is 0, making it sufficient to consider only the second term. To prove that the second term is also 0, we first prove the orthogonality property, i,e.EQUATIONfor any function ϕ.E [E [X G |X T ] ϕ(X T )] = xt E [X G |X T = x t ] ϕ(x t )P (X T = x t ) = xg xt P (X G = x g , X T = x t ) P (X T = x t ) ϕ(x t )P (X T = x t ) = xg xt x g ϕ(x t )P (X G = x g , X T = x t ) = E [X G ϕ(X T )]Therefore, we have the orthogonal property in Eq. ( 16). Now we continue our proof for the main theorem. First, we can decompose Eq. ( 15) as follows:E X G -E [X G |X T ] 2 = E ∥ X G -g A * (X T ) + g A * (X T ) -E [X G |X T ] ∥ 2 = E ∥ X G -g A * (X T )∥ 2 + E [g A * (X T ) -E [X G |X T ] ∥] + 2E X G -g A * (X T ) (g A * (X T ) -E [X G |X T ])Since the gap between pure generative result guided by the training set X T and an ideal text embedding is purely influenced by the training set X T , g A * (X T ) -E [X G |X T ] can also be regarded as a functional over X T , making the last term equal to 0 according to the orthogonal property in Eq. (16) . At this step, we obtain the result in Eq. ( 15).B More Details About CATOD.In this section, we provide more details about our proposed Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD).Initially, we have a large data pool D pool related to the concept given with different quality and begin with an initial randomly-sampled training dataset XT . X T might contain distorted images or even mismatch the given concept, which is common in publicly available datasets. Following the typical paradigm of Active Learning, we train all the models we use on the training set X T . Then, we select a batch of data B from the data pool D pool according to the acquisition functions induced from models. Finally, we move this selected batch of data B to X T , go back to the model training step to re-train or adapt the models, and repeat this cycle until X T reaches its maximum capacity or the quality of models cannot be further enhanced by adding new data. An overall Algorithm for a cycle of CATOD is provided in Alg. 1.Input: Text-to-image model g A (•) that can be integrated with adaptor A, real-world data pool D pool , training data pool X T , budget b, pretrained aesthetic scorer S aes (•), concept-matching scorer S con (•), adaptor A, learning rate group R. Output: Updated training pool X T , updated adaptor A.1: Fine-tune the aesthetic scorer S aes (•, θ) on X T following Sec.B.3; 2: Calculate the aesthetic score γ aes (A) =1 |g A (X T )|xg∈g A (X T ) S aes (x g ) and concept-matching score γ con (A) =1 |g A (X T )| xg∈g A (X T ) S con (x g ) for A; 3: for x ∈ D pool do 4:Calculate the integrated score S(x)(Eq.( 7)); 5: end for 6: B ←{Top-b samples within D pool according to S}; 7: s ← γ(A) (Eq.( 6)); 8: r ← the largest value within R; 9: while r ̸ = min(R) do 10:A ′ ← A -r∇ A x∈X T ∪B L LDM (x, A); 11:Generate a small-scale X G with g(•) conditioned on A ′ ; 12:Calculate score γ(A) for A via Eq.( 6)) and X G ; end if 19: end while 20: X T ← X T ∪ BWe use a weighted-scoring system as the acquisition module that evaluates the quality of images and adaptors in our proposed CATOD framework. In traditional active learning or data selection, the acquisition module is only conducted on samples. However, we observed that an increase in the number of samples does not necessarily contribute to a performance boost like in a traditional active learning setting (which we will verify in Appendix C). Therefore, we propose this weighted scoring system to help schedule adaptor training, which helps locate the best version among them and select samples according to their quality.Our Controllable Adaptor Towards Out-of-Distribution Concepts framework can be divided into 4 steps: (1) Train the scoring system on the current training set X T . (2) Train the adaptor on the training set X T and schedule the training accordingly to the scoring system; (3) Calculate the acquisition score accordingly to the scoring system and the best adaptor for samples in dataset D pool ; (4) Select top-K samples from D pool and move them to train set X T .To give out an quantifiable and unbiased estimation of Eq.( 2), we sample two sets of vectors X = {x 1 , x 2 , . . . , x m } from P and G = {g 1 , g 2 , . . . , g n } from Q, an estimation can be give by:EQUATIONFollowing the description of the aesthetic scorer in Section 3.2, we explain how to personalize it here.Since our primary goal is to make the model distinguish low-quality/high-quality examples for OOD concepts, we design an automatic procedure to assign aesthetic scores to our training set X T at each cycle, based on which we attain an aesthetic training set X aes T . In this work, the aesthetics training comprises three parts: from any adaptors; (3) Randomly sampled set D SI from similar but irrelevant categories. Since we focus on the concepts that the text-to-image model can hardly recognize and visualize, D T G contains lots of plausible, disrupted, or even irrelevant samples, which we assign a zero score. This helps the aesthetic model understand that originally generated samples are low-quality samples. In contrast, D SI somehow describes some attributes for the concept but still does not match the concept, to which we assign the average score of 5.0. We employ the normal regression loss formulated as follows:EQUATIONwhere p denotes score predicted by the original scorer, when p denotes the score we assign, and λ is set to 1.0 in our experiments. Following this paradigm, we train an aesthetic scorer for each category of concepts, and the score personalized for each category from D pool can be predicted accordingly.After selecting top-K data according to this score based on adaptor quality, we assign a lowered aesthetic score to them: px = 10.0 -EQUATIONwhere p x denotes the score given by a generic aesthetic scorer, total represents the number of total active learning cycles, when t current indicates the current cycle. Note that the newly selected images typically have lower quality than those from earlier cycles, we assign progressively lowered scores to them as the learning cycle proceeds.Following the content in the second part "The Active Schedule for Training Adaptors" in Sec. 3.3, we describe our schedule in more detail. In training CATOD, we use this indicator as a signaling proxy throughout the training process. As the number of training samples increases, the quality of selected samples at later cycles might be lower, and a fixed schedule may introduce some unnecessary details within these images (such as watermarks, borders, disturbing objects, etc.). However, these subsequent samples with relatively lower quality do contain some good photographic attributes that can help the embedding evolve. Actually, the potential problems can be alleviated by carefully-designed schedules.An intuitive illustration for this schedule is given in Fig. 6 .We first train the adaptor for 5 epochs, and save an adaptor version A i per 5 epochs, together with its corresponding generated images, constituting a sub-cycle. When this sub-cycle is completed, we filter out the adaptor with the highest quality based on the score γ(A) (Eq.( 6)). If the selected adaptor has the most training epochs across all versions, we keep the learning rate and conduct the next sub-cycle, otherwise, we will reduce the learning rate. Note that if the adaptor quality of this cycle is not as good as that of the previous cycle (because of the quality degradation caused by introducing some unnecessary details), the initial learning rate will be reduced, which helps avoid introducing disturbing elements led by some samples with insufficient quality. The sub-cycles will be continuously conducted until the minimum learning rate is reached or the adaptor quality no longer increases. Our learning rates include 5 × 10 -4 , 2.5 × 10 -4 , 7.5 × 10 -5 , 5 × 10 -5 , 2.5 × 10 -5 .Since the publicly available dataset, i.e., ImageNet, iNaturalist 2018 [67] , IP102 [71] , which we adopt in our research contains 9,244 concepts with 14,883,455 images in total, it is infeasible to manually test and decide what concept is the out-of-distribution concept one by one. Therefore, our procedure for figuring out is two-fold: (1) automatically quantifying the distribution drift of the given concept from what the text-to-image model (Stable Diffusion 2.0, a.k.a SD 2.0) learned with the CMMD metric (thanks to the recent work proposed by Google [27] ); (2) Manually verifying whether the text-to-image model could generate the right content, according to the rank given by the CMMD metric above. Specifically, in the first step, we randomly sample 10 image data for each concept within the datasets and then generate 10 images with the model with the text prompt "A photo of S * ", in which S * denotes the concept name. Then, we calculate the CMMD discrepancy between these two data batches and record them. The CMMD scores of some concepts (including both Out-of-Distribution concepts in the first 5 rows and In-Distribution in the last 5 rows) are listed in Table 5 .Empirically, we observed that SD 2.0 is unable to synthesize the concepts with a CMMD metric above 3.5. Therefore, we pick 25 OOD concepts accordingly divide them into 5 categories, and make our dataset as follows:• Insect: Zopherinae, Antlion, Lycorma Delicatula, Parasitic Wasps, Xylotrechus;• Lizard: Axolotl, Frilled Lizard, Mediterranean House Gecko, Oedura, Opluridae;• Penguin: Emperor Penguin Chick, Gentoo Penguin, King Penguin Chick, Rock Hopper Penguin, Royal Penguin;• Sea Fish: Crampfish, Dragonfish, Garfish, Tigerfish, Tuna;• Snake: Ahaetulla Nasutar, Aipysurus Laevis, Indian Cobra, Pelamis Platurus, Sidewinder.Actually, the dataset we collect is not class-balanced, so the number of samples we can collect varies across different concepts. The statistics of each concept are shown in Table 6 . For the concepts with a D pool less than 1,000 images, we also collect some of them from image host websites including Dreamstime, Flicker, istockphoto, pinterest, and shutterstock. The dataset will be released as soon as the code is released.To verify the effectiveness and extensibility of our proposed CATOD framework, we conduct experiments on 25 different OOD concepts that the large-scale text-to-image (i.e. Stable Diffusion 2.0 [45] ) implement different versions of CATOD. This section explains some important implementation details for the dataset, the aesthetic scorer, the concept-matching scorer we use, and how we design the learning paradigm.The Learning Rate Schedule. As shown in Algorithm 1, we have a learning rate group R, which includes 5 learning rates: 5 × 10 -4 , 2.5 × 10 -4 , 7.5 × 10 -5 , 5 × 10 -5 , 2.5 × 10 -5 . Following the indicator γ(A) in Eq. ( 6), we calculate this value after every epoch. When this indicator falls below the previous evaluation, we reduce the learning rate. If the learning rate cannot be lowered further, we conclude that the model has converged and the training is complete. Note that we choose to train 20 epochs in all experiments as claimed in the main context, it is possible that the training is early stopped before reaching epoch 20.Aesthetic Scorers. To offer the basic knowledge for aesthetic evaluation, we use pre-trained generic models, including ReLIC [81] , TANet [23] , SD-Chad Scorer, and VLAD [59] . Note that the scores given by the generic aesthetic scoring model tend to lie in the majority score range (5 to 6) in our dataset, we ought to personalize this model accordingly to our training set, which we refer to as PIAA [84] . Since PIAA is a typical small sample problem, we adopt similar experimental settings and evaluation criteria by referring to Few-Shot Learning [16] and a previous PIAA research work: PA-IAA [33] .Concept-Matching Scorers. Since the first step of general image retrieval is feature extraction, we use the CLIP ViT-L/14 encoder, which is a commonly used and reliable feature extractor [2, 82, 34] .To further validate the effectiveness of CATOD, we also present compositional results based on the LoRA adaptors we obtained, as illustrated in Figure 7 . We compose different types of concepts, including (i) background concepts such as brick wall and grassland for Emperor Penguin Chicks, LoRA adaptor fully trained on concepts "Frilled Lizard" and "Emperor Penguin Chick". We try to compose these creatures with background elements, in-distribution concepts, and out-of-distribution concepts learned by other adaptors. The final results show high quality with minimal disruptive details. and desk and beach for Frilled Lizard; (ii) in-distribution and common concepts, like the dog; (iii) in-distribution and similar concepts, like Adelie penguin for Emperor Penguin Chicks, and Alligator Lizard for Frilled Lizard; (iii) out-of-distribution concepts from other adaptors, like King Penguin Chicks for Emperor Penguin Chicks and Opluridae for Frilled Lizard. Our observations reveal that the synthetic results on out-of-distribution concepts with CATOD can seamlessly integrate different background elements, even if they were not present in real-world images (columns 1 to 2). For other in-distribution concepts like dogs, LDMs tend to represent a specific species with similar color and texture, while different concepts remain highly distinguishable within one image (columns 3 to 4). For out-of-distribution concepts, we also note that the creatures are depicted correctly, but may exhibit some confused visual details that negatively impact the aesthetics of the image (column 5).C.4 More Analysis. CATOD is more reliable on schedules. From these results, we can draw the following conclusions:• Initial batch size has a more significant impact on randomly initialized samples than on high-quality samples. Specifically, the performance change with high-quality samples is at most 1.40 in the CLIP score and 0.10 in the CMMD score concerning different initial numbers of training data. In contrast, the performance change on low-quality is up to 7.65 in the CLIP score and 0.45 in the CMMD score.• The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.Recall that Stable Diffusion 2.0 fails to generate emperor penguin chicks, which accounts for the fact that adult emperor penguin chicks and their chicks have different appearances. We raise another question: can we train an adaptor with a concept that has different shapes? Inspired by this, we also test our proposed CATOD on the concept "lycorma delicatula", which is a kind of pest with 3 different forms in its life-cycle, and surprisingly find that with 300 training samples, the best version of adaptors successfully produces all these 3 shapes, as shown in Figure 9 (b). These results further validate the effectiveness of our framework, even in the presence of appearance ambiguity.Our experiments in the main content are all conducted on Stable Diffusion 2.0 (SD 2.0). To further validate the superiority of our proposed CATOD, we conduct additional experiments on other architectures (based on LoRA), including Stable Diffusion 1.5 (SD 1.5) and Stable Diffusion XL (SDXL). The results are shown in Table 9 .This table shows that our proposed CATOD is also compatible with different architectures with notable performance gain compared to the baselines. 9 . In this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity.Since the training set contains samples with different angles in CATOD, it is also easy to produce objects with different angles, and we show the examples on concepts "Axolotl" and "Emperor Penguin Chick" in Figure 9 . We can observe that training samples with different angles selected by CATOD also lead to diverse angle in their generative results.NeurIPS Paper ChecklistJustification: There is no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: MIT License has been added already.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 


Decision: The paper aims to improve the adaptation of large-scale text-to-image generative models that allows them to accurately depict the visual
details of out-of-distribution concepts. Inspired by the empirical evidence that the failure cases usually correlate with the low quality of training data, the paper proposes to leverage an active learning paradigm which couples high-quality data accumulation and
adaptor training, enabling a finer-grained enhancement of generative results. A novel weighted scoring system is developed to automatically balance between aesthetics and concept-matching, the two key factors that impact the quality of synthetic results. The overall contribution is supported by both theoretical analysis and empirical results.

Reviewers provided constructive feedback that helped the authors further improve the quality of the paper though the rebuttal process. At the end of the reviewer-author discussion period, all reviewers were in favor of accepting the paper. The authors are encouraged to incorporate the reviewers' feedback into the final version of the paper.

Reviews: review: paper_summary: paper_summary: This work presents CATOD, a new method for dealing with the challenging scenario of adapting generative models to OOD concepts. In particular, CATOD leverages an active learning setting to update the adaptor to generate better OOD images more broadly. Furthermore, the authors provide both extensive empirical evaluations and theoretical analysis which further support the validity of the CATOD methodology.
strength_weakness: strength_weakness: The primary concern with this paper is the sparse empirical evaluations presented for CATOD. In particular, the 25 OOD concepts chosen seem few with only a small set of 100 samples left out for validation.
questions: questions: The reviewer would like to have further clarification regarding the experimental setup of the paper. In particular, the reviewer would like to hear the author more directly address whether the small evaluation set used for the experiments is a reasonable concern.
limitations: limitations: The authors don't provide any full discussion of the limitations of the proposed methodology. Additionally, no direct comment has been made regarding societal impacts, however, the reviewer does not see any potential negative impacts derived from this work.
paper_summary: paper_summary: This work tackles the challenge of adapting text-to-image diffusion models to out-of-distribution (OOD) concepts. The authors introduce a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD), which employs an active learning paradigm to iteratively accumulate high-quality training data and optimize adaptor training. The framework features a dual scoring system that balances aesthetics and concept matching, ensuring improved quality and conciseness of the generated images.
strength_weakness: strength_weakness: 1.	The key contribution of this paper is Equation (6) and (7), which are claimed to be use as a signal to reduce the learning rate, stop the training in time, and select samples for the next cycle.  It seems the innovation is that the authors identify two relevant factors and utilize the strategy of active learning. Am I correct? 
2.	Authors use MMSE to reinterpret the loss function of the LDM and decompose it into two terms (Eq. 10). As the description in lines 213 to 220 of the paper, associating these two terms directly with aesthetic preservation and concept-matching score is rather far-fetched and vague. Kindly advise if I am wrong.
3.	If the aesthetic and concept-matching scores are really that important, why just use those two scores to pick suitable samples for adaptation from the beginning? This would also avoid the later calculations.
questions: questions: 1. Is the CMMD shown in Figure 1 calculated using Real Images and before adapting images?
2. Are the high-quality samples shown in Figure 2 generated by vanilla Textual Inversion, DreamBooth, and LoRA? Or are they already using the CATOD framework?
3. Is “if r_k is the closet representation for r_k” in line 145 of the paper correct?
4. Eq. 6 and the description in lines 173 to 174 of the paper confuse me. Shouldn’t the maximum value of \gamma (A) be \gamma_aes (A)=\gamma_con (A)=10+40k, k=0...n, and the minimum value be \gamma_aes (A)=30+40k, or \gamma_con (A)=30+40k, k=0...n?
5. Associating the two terms of Eq. 10 directly with aesthetic and concept-matching scores requires further theoretical explanation and empirical analysis.
6. I noticed in the experiment section that the training period is about 20 epochs. Would this somehow be too less? If there were enough training cycles, we would get an overfitted LDM capable of generating images containing specific OOD concepts.
7. In Algorithm 1, lines 9 to 19, it does not look like new training samples are being selected for each epoch.
8. Does the learning rate set R determine the stopping of the training? How would you guarantee that the model has converged at this point? As question 5, a sufficiently overfitted LDM should be able to generate these OOD concepts.
9. This paper tests the proposed CATOD on SD 2.0. Have you conducted experiments to evaluate its performance on other text-to-image models, such as SD 1.5 and SDXL? Including these results would help verify the versatility of your method.
10. I am curious about how CATOD performs with different initial training set sizes (e.g., 10 and 50) and varying quality. Adding this as an ablation experiment would provide valuable insights into the robustness of your method.
11. I am also interested in whether similar categories (e.g., other kinds of penguins) might be distorted after adaptation. Have you conducted any experiments to evaluate the degree of distortion in these cases?
12. Since CMMD is a newly proposed metric by Google in CVPR 2024, it would be helpful to list the advantages of using CMMD compared to traditional metrics like Inception Score, FID, and MMD. This would help readers understand the rationale behind choosing CMMD.
limitations: limitations: n/a
paper_summary: paper_summary: This paper introduces a novel framework called Controllable Adaptor Towards Out-of-Distribution (OOD) Concepts (CATOD). CATOD is designed to adapt text-to-image models to OOD concepts and generate high-quality images of those OOD concepts accordingly. The authors identified the challenge of accurately depicting OOD concepts due to low-quality training data. CATOD employs an active learning approach to iteratively accumulate high-quality training data and update the adaptor. The framework incorporates an aesthetic score and a concept-matching score to guide the accumulation of training data. Extensive experiments demonstrate significant improvements in both the CLIP score and the CMMD metric.
strength_weakness: strength_weakness: 1. To support the claim in Line 189 that your selection "guarantees the sample diversity", the authors should report the diversity metrics of the generated samples. 
2. The paper provides ablation results on the design of weighted scoring strategies, which is valuable. I think more extensive ablation studies on the impact of initial data size, the type of text-to-image encoders, etc. can help signify the paper's contribution.
3. To further investigate how the weighting strategy work in your proposed CATOD, it would better to list how the weights of aesthetic/concept-matching scores change as your learning cycle proceeds over different concepts.
questions: questions: 1. Following weakness 1, can you compare the diversity of the fine-tuned model under different strategies?
2. It would be intriguing to investigate how initial data size impact the experiments, especially for the multiple concept scenario as you mentioned in Sec. 5.2 following weakness 2.
3. For different concepts, which of the two scores gets high weightage (in equation 11) as the training progresses?
limitations: limitations: See weakness.
paper_summary: paper_summary: The paper introduces CATOD, a framework designed to enhance the adaptation of text-to-image models for out-of-distribution (OOD) concepts. It addresses the issue of low-quality training data by employing an active learning approach that iteratively improves the training set. The framework utilizes a scoring system comprising aesthetic and concept-matching scores to guide data selection and training. This scoring system dynamically determines the relative weight of the aesthetic score and content matching score to prioritize samples for selecting the training set. The authors demonstrate significant performance improvements using CATOD, achieving notable improvements on standard metrics.
strength_weakness: strength_weakness: 1. The paper does not provide enough detail to clarify how the CMMD metric differs from other commonly-used metrics for image generation, such as Inception Score and FID. The authors should further explain this.
2. Since your proposed CATOD is trained on few samples (100 for single-concept and 200 for multi-concept), the generation model may be biased toward specific samples. Reporting diversity metrics of the generated samples can help clarify this issue.
questions: questions: 1. I wonder if your proposed CATOD also outperforms other methods on previous metrics such as FID and the Inception Score.
2. In the paper, random sampling and CLIP-score-based sampling are used as the comparison method, and finally 100-200 images are selected for training. Would it be better to fine-tune the model directly using all 1000 images?
3. In addition to the three methods mentioned in the paper, are there other custom content generation methods that your proposed CATOD framework is still applicable to, such as LyCORIS[1] and ELITE[2]?

[1] YEH, SHIH-YING, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. "Navigating text-to-image customization: From lycoris fine-tuning to model evaluation." In *The Twelfth International Conference on Learning Representations*. 2023.

[2] Wei, Yuxiang, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. "Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation." In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 15943-15953. 2023.
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: Post-rebuttal comments.
comment: The authors have addressed my concerns. 
I keep positive attitude towards this paper and increase my score by one.
**Part 3: Using 1000 data is not feasible (Q2)**

Using all the 1000 data for training is not feasible for adapting OOD concepts. The experimental results of using all data are as follows:

| Combinations          | Axolotl (CLIP$\uparrow$) | Axolotl (CMMD$\downarrow$) | Emperor Penguin Chick (CLIP$\uparrow$) | Emperor Penguin Chick (CMMD$\downarrow$) |
| --------------------- | ------------------------ | -------------------------- | -------------------------------------- | ---------------------------------------- |
| DreamBooth + RAND     | 66.19                    | 1.13                       | 67.34                                  | 1.54                                     |
| DreamBooth + ALL DATA | 68.24                    | 1.01                       | 70.12                                  | 1.23                                     |
| DreamBooth + CATOD    | 72.25                    | 0.88                       | 74.38                                  | 0.83                                     |
| TI + RAND             | 59.67                    | 2.11                       | 62.89                                  | 2.10                                     |
| TI + ALL DATA         | 52.79                    | 3.04                       | 55.63                                  | 2.67                                     |
| TI + CATOD            | 69.95                    | 1.49                       | 68.14                                  | 1.15                                     |
| LoRA + RAND           | 65.35                    | 1.41                       | 68.23                                  | 1.58                                     |
| LoRA + ALL DATA       | 67.24                    | 1.47                       | 71.17                                  | 1.33                                     |
| LoRA + CATOD          | 72.87                    | 1.37                       | 73.19                                  | 1.13                                     |

As we see in this table, performing adaptation over the full dataset can lead to severe performance loss compared to using carefully selected ones. On Textual Inversion (TI), it even becomes worse than random sampling. This is due to that the full dataset contains lots of "bad" samples, which exhibit too many disruptive elements -- in turn -- introducing wrong details that mislead the concept adaptation.

**Part 4: Generalizing CATOD to other customization methods. (Q3)** 

Thanks for the advice! CATOD can be easily generalized to ELITE and LyCORIS since it is designed as a general framework for the adapters. We show the results for applying CATOD to ELITE/LyCORIS on concepts "axolotl" and "emperor penguin chick" as follows:

| Combinations    | Axolotl (CLIP) | Axolotl (CMMD) | Emperor Penguin Chick (CLIP) | Emperor Penguin Chick (CMMD) |
| --------------- | -------------- | -------------- | ---------------------------- | ---------------------------- |
| LoRA + CLIP     | 72.87          | 1.37           | 73.19                        | 1.13                         |
| ELITE + CLIP    | 74.14          | 1.03           | 74.37                        | 1.01                         |
| LyCORIS + CLIP  | 72.95          | 1.18           | 74.08                        | 1.08                         |
| LoRA + CATOD    | 75.35          | 0.85           | 74.89                        | 0.79                         |
| ELITE + CATOD   | 75.68          | 0.83           | 74.68                        | 0.74                         |
| LyCORIS + CATOD | 76.04          | 0.79           | 75.05                        | 0.73                         |

From these results, we can also see that CATOD gives a consistent boost to ELITE and LyCORIS compared to other sampling strategies. Despite that CATOD is compatible with ELITE and LyCORIS, they do not exhibit a notable performance gain compared to LoRA.
**Part 7: Why CMMD is a good metric (Q12)**

As reported in many recent works [1,2,3], most popular image matching metrics like Inception Score, Precision/Recall and FID (Frechet Inception Distance) may disagree with human raters, thus ill-suited for evaluating recent text-to-image models. In comparison, CMMD uses CLIP embeddings and Maximum Mean Discerpancy (MMD) that correctly ranks the image sets based on the severity of the distortion. The limitations of FID and previous metrics can be listed as follows:

|                      | FID and other previous metrics                               | MMD distance                                                 |
| -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |
| Inception embeddings | × Weak image embeddings<br />× Normality assumption<br />× Sample inefficient<br />× Biased estimator | × Weak image embeddings<br />√  Distribution-free<br />√  Sample inefficient<br />√  Unbiased estimator |
| CLIP embeddings      | √ Rich image embeddings<br />× Normality assumption<br />× Sample inefficient<br />× Biased estimator | √  Rich image embeddings<br />√  Distribution-free<br />√  Sample inefficient<br />√  Unbiased estimator |

[1] Jayasumana, Sadeep, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. "Rethinking fid: Towards a better evaluation metric for image generation." *arXiv preprint arXiv:2401.09603* (2023).

[2] Grimal, Paul, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. "TIAM-A Metric for Evaluating Alignment in Text-to-Image Generation." In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 2890-2899. 2024.

[3] Lee, Tony, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang et al. "Holistic evaluation of text-to-image models." *Advances in Neural Information Processing Systems* 36 (2024).

**Part 8: Other minor questions**

**Q1 (About Figure 1):** Yes, the CMMD in Figure 1 is calculated before adapting. This is used to show how out-of-distribution a concept is before adapting.

**Q2 (About Figure 2):** Yes, the high-quality samples in the third part of Figure 2 are selected by the CATOD framework.

**Q4 (About Eq. 6):** Apologies for the confusion! Since both the aesthetic and concept-matching scores range from 0 to 10, Equation 6 monotonically increases according to both variables, with both sine functions ranging from 0 to 1. We will ensure that the scores $\gamma_{aes}$ and $\gamma_{con}$ are clearly stated to range from 0 to 10 in future revisions.

**Q6 (About the epochs to use):** Training for 20 epochs is common for adaptors such as LoRA, DreamBooth, and other related methods, as they typically require much less data (usually around 100 samples per concept). Training for too many epochs can lead to the underlying model replicating the images as exact copies.

**Q7 (About algorithm 1):** We don't need to select new samples at each training epoch. The selection is performed in lines 2-6, prior to training. The algorithm demonstrates the process of a single Active Learning cycle, while training CATOD involves several cycles. The selection is conducted at the beginning of each cycle.

**Q8 (About the schedule):** R includes 5 learning rates: $5\times 10^{-4},2.5\times  10^{-4},7.5\times10^{-5},5\times 10^{-5},2.5\times 10^{-5}$. Following Q7, we calculate the indicator $\gamma(A)$ after every epoch. When this indicator falls below the previous evaluation, we reduce the learning rate. If the learning rate cannot be lowered further, we conclude that the model has converged and the training is complete.

**Part 9: Some typo errors  (According to Q3)**

Thanks for pointing out the mistake! In lines 173 and 174, the sentence should be "if $r_k$ is the closet representation for $r_x$".
**Part 5: The impact of initial size/quality in CATOD (Q10)**

Thanks for the advice! It is intriguing to ablate CATOD over the initial training set size/quality. In detail, we retest CATOD with different numbers of initial samples (10,20,50) and different quality (high-quality and random sampling), with 100 images sampled at last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results are shown as follows:

| Initial Setting         | Axolotl (CLIP) | Axolotl (CMMD) | Emperor Penguin Chick (CLIP) | Emperor Penguin Chick (CMMD) |
| ----------------------- | -------------- | -------------- | ---------------------------- | ---------------------------- |
| 10 HQ initial samples   | 73.99          | 0.84           | 74.97                        | 0.71                         |
| 10 RAND initial samples | 72.89          | 0.93           | 73.95                        | 0.75                         |
| 20 HQ initial samples   | 75.35          | 0.82           | 74.89                        | 0.69                         |
| 20 RAND initial samples | 72.27          | 1.10           | 72.38                        | 0.73                         |
| 50 HQ initial samples   | 75.37          | 0.81           | 74.56                        | 0.63                         |
| 50 RAND initial samples | 70.39          | 1.34           | 68.85                        | 1.12                         |

From these results, we may draw the following conclusions:

1. With HQ initial samples, the initial batch size does not have a significant compact, since a change in this size is not more than 0.3 in the CLIP score and 0.08 in the MMD score. However, this initial size has a significant impact with randomly initialized samples with at most a 5.71 loss in CLIP score and 0.49 in CMMD. We account this phenomenon for that randomly initialized images contain more bad-quality ones that mislead the adaptation.
2. The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.

The full results will be added in our revisions.

**Part 6: Evaluating the quality of ID concepts after adapting OOD concepts. (Q11)**

In brief, our method will not cause much degradation in non-OOD concepts. To prove this, we list a comparison of the performance of the following non-OOD concepts (corresponding to Q1) over SD 2.0 when fine-tuning with LoRA. The results are shown in the table below:

| Insect            | Thrips | Flea Beetle | Aphids | Red Spider | Meadow Moth |
| ----------------- | ------ | ----------- | ------ | ---------- | ----------- |
| **CLIP (before)** | 75.35  | 72.76       | 75.59  | 74.27      | 73.93       |
| **CLIP (after)**  | 75.17  | 71.89       | 75.76  | 74.25      | 73.78       |
| **CMMD (before)** | 2.05   | 1.28        | 1.16   | 1.85       | 1.47        |
| **CMMD (after)**  | 2.08   | 1.32        | 1.30   | 1.93       | 1.49        |

| Penguin           | Emperor Penguin | King Penguin | Little penguin | Magellanic Penguin | Adelie Penguin |
| ----------------- | --------------- | ------------ | -------------- | ------------------ | -------------- |
| **CLIP (before)** | 76.89           | 79.23        | 80.45          | 75.45              | 73.68          |
| **CLIP (after)**  | 76.56           | 78.97        | 79.78          | 75.42              | 73.55          |
| **CMMD (before)** | 1.19            | 0.75         | 1.46           | 1.60               | 0.77           |
| **CMMD (after)**  | 1.26            | 0.80         | 1.58           | 1.61               | 0.86           |

From this table, we can see that our proposed CATOD only leads to a performance loss of at most 0.15 on the CLIP score and at most 0.12 on CMMD on the non-OOD concepts after fine-tuning. This proves that our method does not distort non-OOD concepts when adapting native models to OOD concepts.