Title: Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Genearation

Text-to-Image

Diffusion

Out-of-Distribution

Active Learning
Abstract: The recent large-scale text-to-image generative models have attained unprecedented performance, while people established *adaptor* modules like LoRA and DreamBooth to extend this performance to even more unseen concept tokens. However, we empirically find that this workflow often fails to accurately depict the *out-of-distribution* concepts. This failure is highly related to the low quality of training data. To resolve this, we present a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD). Our framework follows the active learning paradigm which includes high-quality data accumulation and adaptor training, enabling a finer-grained enhancement of generative results. The *aesthetics* score and *concept-matching* score are two major factors that impact the quality of synthetic results. One key component of CATOD is the weighted scoring system that automatically balances between these two scores and we also offer comprehensive theoretical analysis for this point. Then, it determines how to select data and schedule the adaptor training based on this scoring system. The extensive results show that CATOD significantly outperforms the prior approaches with an 11.10 boost on the CLIP score and a 33.08% decrease on the CMMD metric.

Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild

Abstract

The recent large-scale text-to-image generative models have attained unprecedented performance, while people established adaptor modules like LoRA and DreamBooth to extend this performance to even more unseen concept tokens. However, we empirically find that this workflow often fails to accurately depict the out-of-distribution concepts. This failure is highly related to the low quality of training data. To resolve this, we present a framework called C ontrollable A daptor T owards O ut-ofD istribution Concepts (CATOD). Our framework follows the active learning paradigm which includes high-quality data accumulation and adaptor training, enabling a finer-grained enhancement of generative results. The aesthetics score and concept-matching score are two major factors that impact the quality of synthetic results. One key component of CATOD is the weighted scoring system that automatically balances between these two scores and we also offer comprehensive theoretical analysis for this point. Then, it determines how to select data and schedule the adaptor training based on this scoring system. The extensive results show that CATOD significantly outperforms the prior approaches with an 11.10 boost on the CLIP score and a 33.08% decrease on the CMMD metric.

1 Introduction

The generative modeling for text-to-image has attained unprecedented performance most recently [37, 45, 41, 49]. Notably, by training over billions of text-image data pairs [52, 51], the family of diffusion models has allowed high-fidelity image synthesis directed by the prompt provided in production. Despite their massive successes, these models still fail to generate images with decent quality and matching semantics, when encountering prompts that contain unseen or out-of-distribution concept tokens [60, 12, 21]. Simply put, the reason causing this failure is due to that the training set is not unbounded with limited variations. On the side of production, this limitation could significantly impact the practicality of this technique in real-world applications.

To deal with such concepts, recent works have resorted to adaptors such as Textual Inversion [18], DreamBooth [47, 58, 48], and LoRA [25, 66, 76], which tunes only a small part of the text-to-image model or insert extra modules. These adaptors largely reduce the training costs, and more importantly, preserve the visual aesthetic information originally learned by the underlying model.

1 , 2

Real

Image

Before

Adapting

After

Adapting

(LoRA)

Axolotl

(CMMD: 4.24)

(CMMD: 3.42)

Alligator Lizard

(CMMD: 1.44)

Agama

(CMMD: 2.69)

Whiptail

(CMMD: 0.88)

Emperor

(CMMD: 4.14)

King

(CMMD: 3.68)

RockHopper

Penguin

(CMMD: 4.04)

Magellanic

Penguin

(CMMD: 1.60)

Adelie

Penguin

(CMMD: 0.77)

However, in this paper, we have found that recent works still struggle to accurately depict the visual details of out-of-distribution concepts (with a CMMD score above 3.5), as illustrated in Figure 1. Adapters like LoRA are able to accurately represent the shape and color of OOD concepts compared to the generative results before adaptor training, but they fall short when it comes to finer details such as texture, contours, and patterns. This issue arises from the fact that current studies predominantly focus on variations of in-distribution (ID) concepts (e.g., humans, dogs, and cats) while ignoring the out-of-distribution (OOD) ones. This failure motivates us to think about what makes the problem of distorted visual details happen when training adaptors.

In Figure 2, we observed that how an adaptor depicts OOD concepts can be significantly influenced by the quality of the training data. If the model is trained on samples containing disruptive objects, the resulting generative outputs are likely to reflect these disruptive elements. When the training data contains images with vague or very small instances of the OOD concept, the generative results may appear low-quality. In contrast, the high-quality data for adapting OOD concepts usually contains a single and clear object corresponding to the given concept, which is highly distinguishable from the background and other types of objects and helps produce accurate results with high-fidelity. However, manually picking such high-quality data requires much human labor and expertise, which may crucially limit the versatility of text-to-image models. Therefore, an effective paradigm to locate high-quality samples of OOD concepts is important for the practical shipping of this field.

To this end, we developed a framework called C ontrollable A daptor T owards O ut-ofD istribution Concepts (referred to as CATOD) which aims to identify high-quality samples to guide the adaptor training. This framework follows the Active Learning (AL) paradigm [54, 44], involving iteratively accumulating training data and updating the adaptor. The profound motivation of this approach is to comprehensively model the interaction between training data and the underlying text-to-image model. Specifically, CATOD includes two interconnected scores: the aesthetic score and the conceptmatching score, following the observation that object clarity and uniqueness largely impact generative results, as illustrated in Figure 2. Based on this, we devised a weighted scoring system that adapts itself according to the adaptor to select high-quality data while also properly balancing the two scores. With the carefully selected high-quality data, we schedule the adaptor training based on the quality of generative results evaluated through this scoring system.

In summary, our contributions are as follows: (i)-We have identified the challenge of adapting text-to-image models to out-of-distribution (OOD) concepts, where recent studies often struggle to accurately depict them; (ii)-We have introduced a framework called CATOD that iteratively updates training data and the adaptor to generate OOD concepts precisely; (iii)-Our extensive experiments verified that CATOD achieves significant performance gain with up to 11.10 on the CLIP score and 33.08% on the CMMD metric; (iv)-We have also offered theoretical insights into the key factors: aesthetics and concept-matching , which contribute to the effectiveness of our method.

(Disruptive Objects)

Textual Inversion

DreamBooth

LoRA

(Small Objects)

Textual Inversion

DreamBooth

LoRA

High-Quality

Samples

Textual Inversion

DreamBooth

LoRA

2 OODProblem in Latent Diffusion Models

Revisiting Latent Diffusion Models (LDMs). Latent Diffusion Models (LDMs) [45] comprise two components: a diffusion process operating the latent space and an auto-encoder which contains an encoder E mapping an image into the latent space and a decoder D that reconstruct images from latent codes. Furthermore, the diffusion process can be conditioned on the output of text embedding models, enabling the auto-encoder to integrate the information derived from texts. Let x be the image, the CLIP textual encoder c θ that maps the corresponding text y into the latent space, the LDM loss is:

where t denotes the time step, z t denotes the latent code noised at time step t , when ϵ, ϵ θ represents the noised samples and the denoising U-Net [46], respectively. Through this noising-denoising procedure applied to the latent codes, LDM enables the underlying model to integrate information derived from texts into the visual results, while also allowing more flexibility to produce images.

The OOD Concepts for LDMs. Intuitively, out-of-distribution (OOD) concepts refer to the category of data whose distribution deviates significantly from what the model has learned. This degree of drifting can be quantified by an MMD score [27], which evaluates the discrepancy between ground-truth images and the generative results in the image latent space. Formally, for two probability distributions P and Q , the MMD distance with respect to a positive definite kernel k is:

where x p , x ′ p independently follow the distribution P while x q , x ′ q independently follow the distribution Q , with k the Gaussian RBF kernel [17]. In our implementation, we sample two sets of vectors from the distribution P and Q , then use CLIP embeddings [40] to calculate this score, which is also named as CMMD [27]. We set a concept with a high CMMD score (above 3.5) as an OOD concept.

OODConcepts are Hard to Adapt. Recent text-to-image LDMs [45, 39] have achieved unprecedented performance on a wide range of concept tokens. However, we have found that there still exist many concepts that make LDMs fail after the adapter is fully trained. As we show in Figure 1, there are several discoveries: (i)-The concepts with higher CMMD scores are much more challenging for the underlying model to generate or adapt. The concepts with a CMMD score above 3.0 show explicit wrong visual details. For Axolotl and Frilled Lizards with CMMD above 4.0, the LDMs even generate the wrong species; (ii)-A higher CMMD score indicates a more severe loss of visual details when training adaptors. The concepts with CMMD scores above 3.5 in Axolotls and Emperor Penguin Chicks, show explicit distorted visual details, like color, texture, and delicate details. For Axolotl, the generative results show a wrong number and wrong positions of amateurs. For Emperor Penguin Chicks, the generative results show the wrong fur color of their heads and wings. To summarize, the higher the CMMD score of a concept, the more difficult it is for LDMs to adapt.

The High-Quality Matters. We further observe that the generative results are quite sensitive to training data when training adaptors meet OOD concepts. As shown in Figure 2, when the training images contain disruptive elements, the visual features of these disruptive elements will be easily introduced into the generative results. For example, when an adult emperor penguin appears in training data, then the black fur on their back can easily appear when generating their chicks, despite that their chicks have white fluffs as shown in the left part of Figure 2. If the object of the desired

Photo of a

S* penguin

Sample

Score

Aesthetic Score

Concept Matching Score

CLIP Model

Diffusion Model

Adaptor Module

?

?

?

?

Scoring System

Data Space

Fine-tune

Query Score

Weighted

Scorer

Selected Training

Samples

Next Cycle

Selected Samples

from Last Cycle

concept appears to be too small within the image, then the generative results tend to be bad-looking, since the necessary visual details are not fully identified. In the middle part of Figure 2, we can see that distorted visual details like texture and shape in axolotl and emperor penguin chicks largely harm the aesthetics of the generated image. To generate images correctly and good-looking, we require enough amount of images with objects of high fidelity and do not contain disruptive elements, namely High-Quality samples as shown in the right part of Figure 2. Therefore, we aim to devise an effective data selection strategy for locating those high-quality images for these OOD concepts.

3 Method

3.1 Overall Architecture

We aim to adapt OOD concepts to LDMs correctly by an iterative data selection criterion that locates high-quality data and training adaptors accordingly as shown in Figure 3. Consequently, our method would alleviate issues introduced by disruptive elements, e.g. , irrelevant objects, and blurry images, while maintaining high fidelity of generative results. The core to CATOD is a scoring system, which consists of an aesthetic scorer and a concept-matching scorer, aiming to resolve the problems of incorrectly introduced visual details and distorted objects we have observed in Section 2. By properly trading off between these two scores, CATOD locates the most valuable samples for adapting underlying LDMs to OOD concepts and making the desired OOD concept depicted correctly.

3.2 The Scoring System

As described above, we need to estimate the potential impact of real-world samples over LDMs, according to which we select the most high-quality samples for training. In Section 2, we have observed two major factors that significantly impact generated image quality: object clarity and disruptive elements. Diving into the loss term L LDM ( x, y ) in Eq. (1), we may also observe that the training set X T should be optimized towards both the underlying model (including the embedded adaptor) and the conditional text y , which indicates two important factors: object clarity and conceptmatching achieved by preserving aesthetic information originally learned by LDMs and accurate image-text matching, respectively. Therefore, we attribute the quality of images to aesthetic and concept-matching and design two decoupled scorers accordingly.

The Aesthetic Scorer. Aesthetic evaluation is a long-standing field, which comprehensively considers whether the lighting, contrast, texture, and other photographic factors of an image are consistent with

human aesthetics. The general aesthetic scorer can be simply described as p = S aes ( x ) , where S aes indicates the aesthetic scoring model, p denotes the predicted score, and x represents the input image. Following the work of PA-IAA [33], we fine-tune the generic aesthetic model and make it adapt to OOD concepts with personalized preference scores. In general, we assign a high score to the samples within the training set X T , assign a low score to samples from irrelevant categories and samples generated by underlying LDMs without an adaptor, and use them to fine-tune the aesthetic scorer. More details of this personalization are given in the Appendix B.3.

The Concept-Matching Scorer. Intuitively, concept-matching describes whether the OOD concepts get perfectly reflected in the generated results. A similar task is image retrieval, which is designed to retrieve images containing objects describing the desired concepts. Since image retrieval also relies on feature maps and some sort of matching score to retrieve images, we adopt matching score from VLAD-related works [28, 64, 55] as follows:

where ϕ ( r x , r k ) = 1 if r k is the closet representation for r x and is set to 0 otherwise. Simply put, VLAD regards the extracted representations for X T as a codebook and maps each image to its nearest code. Note that samples within X T mostly consist of clean and clear samples, this score is sufficient to quantify whether the object if exists in the given image distinguishes itself from other photographic components and matches the given concept.

3.3 Active Data Acquisition

Optimization with Active Learning (AL). Aiming to mitigate the problems caused by low-quality training samples, we proactively integrate the training data X T into our objective as follows:

Since the optimal set is initially unknown, a one-step optimization can easily lead to convergence to local optima. Therefore, we use an iterative paradigm to optimize adaptor A and training data X T , respectively. In our implementation, we adopt the paradigm of AL [44, 54] to perform the optimization of X ( t ) T by data accumulation before training adaptors, with t denotes the time step:

where b is the number of samples added to the training pool at each cycle. The main reason for using AL is its preferred sample efficiency, with better controllable data bias management [15, 50]. Instead of repeatedly selecting data from the whole real-world data pool, AL provides a more efficient procedure to optimize training data by using data selection to accumulate high-quality training data. To this stage, the learning procedure of CATOD is relatively clear: the sample pool is progressively accumulated -by Eq. (5) -and the optimization of the adaptor A is straightforward (Fig. 3).

Remember that AL involves iteratively updating the adaptor and the training data, it is important to design a training schedule for adaptors and determine how to acquire high quality based on the two scorers mentioned above. The primary objective of this design is to achieve a dynamic trade-off between the two scores. The specific details of these designs are described below.

The Active Schedule for Training Adaptors. The training schedule has been found crucial for successful adaption [47, 74, 65]. Since our training data continuously expands as the learning cycle of AL proceeds, the training schedule will be even more important. To arrange this schedule, we first calculate the aesthetics score γ aes ( A ) = 1 | g A ( X T ) | ∑ x g ∈ g A ( X T ) S aes ( x g ) and concept-matching score γ con ( A ) = 1 | g A ( X T ) | ∑ x g ∈ g A ( X T ) S con ( x g ) of the adaptor A based on its generative results g A ( X T ) , in which both the aesthetic score γ aes ( A ) and γ con ( A ) range from 0 to 10. Then, we use a trigonometric indicator that comprehensively measures its performance:

Notably, γ ( A ) peaks when the two scores get close or around the common value of 5.0 for most samples. Meanwhile, it bottoms when the scoring exhibits a stronger signal of being biased (to either

side). The properties emphasize that adapters balancing the two factors are of better quality. In training CATOD, we use γ ( A ) as a signal to reduce the learning rate and stop training in time.

Trading-off the Two Scores in Data Acquisition. After getting the adaptor A , we continue to select the most suitable samples for the next-cycle training. Notice that whether newly selected images enhance the adaptor depends on both aesthetics and concept-matching, we ought to adjust our preferences according to the adaptor. In more detail, when the adaptor has high aesthetics but does not accurately depict the OOD concept, samples with high concept-matching scores better enhance the adaptor; when the adaptor fails to exhibit photographic attributes consistent with humans, samples with high aesthetics are preferred. To implement this preference, our acquisition score is:

This formulation offers some meritable advantages. On one hand, the balancing terms are not prefixed or manually tuned, but dynamically dependent on the scores marginalized over the current generations. Further, when the score of either side gets larger, the corresponding balancing coefficient, in turn, decreases, thus attaining a proper trade-off. As a result, this mechanism encourages an alternation of sample selection towards both scoring metrics by selecting TopK samples to add to the training data X T accordingly, which effectively guarantees the sample set diversity.

4 Theoretical Insights

This section presents our theoretical analyses of why aesthetic/concept-matching scores work in OOD Adaption. Specifically, we derive the distance between real-world data distributions and synthetic data distributions and then induce the important factors that affect this distance. We first introduce the minimum mean square error (MMSE) [22, 11, 6] to measure the discrepancy between distributions:

Definition 4.1. The minimum mean square error (MMSE) of estimating an input random vector ̂ X ∈ R n from an observation/output X ∈ R k is defined as

in which M ( R n ) denotes the space consisting of all measurable functions on R n .

Notice that we are trying to produce the best generation results which are initially unknown, our adaption task can be also regarded as estimating the optimal distribution with carefully selected data. By denoting the ideal generative results and the ideal adaptor with random variables ̂ X G and A ∗ , respectively, we conclude that the LDM loss L LDM is consistent with this MMSE term:

Theorem 4.2. Let L LDM be the LDM loss following Eq. 1, and the image space lies within R n . Then there always exists an ideal random vector ̂ X G ∈ R n and an adaptor A ∗ for LDM, such that

Based on the preceding deduction, we have confirmed that the change in MMSE can also indicate the change in LDMs. To dive deep into the MMSE term, we further decompose it as follows:

Theorem 4.3. (Pythagorean Theorem for MMSE [13].) Following Theorem 4.2, by setting f to the generative model g A , the MMSE term in Eq. (8) can be decomposed into two terms as follows:

in which A ∗ denotes a potential ideal adaptor containing information for text y . Notice that the generative results X G relies on the training set X T , g A ( X T ) can also be written as X G | X T .

Upon revisiting the two terms, we can gain a deeper understanding of the relationship between MMSE and aesthetic/concept-matching score: (i)-The first term is focused on estimating the difference between the generative distribution X G | X T and the ideal distribution of ̂ X G . This assessment helps

us understand how the introduced samples lose visual information within the underlying LDM, thus we can connect this term to aesthetic preservation, i.e. aesthetic score; (ii)-The second term illustrates the degree to which the training set X T distorts the OOD concept information within the ideal adaptor. Hence, a concept-matching score accurately portrays how newly given samples impact this term. At this stage, we have completed the theoretical support showing that both aesthetics and concept-matching are major factors that influence the performance of adaptors.

5 Experiments

In this section, we present the main experimental results both qualitatively and quantitatively. To evaluate our proposed CATOD, we combine it with several works for adaption, i.e. DreamBooth [47], Textual Inversion [18], and LoRA [25]. More experimental results can be found in the Appendix. The source code is attached in the Supplementary.

Datasets. We test our method on datasets with 25 OOD concepts that can hardly be generated through prompt engineering on the text-to-image model. This dataset consists of 5 categories: insect, lizard, penguin, seafish, and snake, and each category contains data from 5 OOD concepts. Each concept has 1,000 examples in total with 100 samples left out for validation. The dataset is collected from publicly available datasets including ImageNet, iNaturalist 2018 [67], IP102 [71].

Implementation Details. We conduct the active generation experiments on our proposed CATOD and three representative adaptors, i.e. DreamBooth [47], Textual Inversion [18] (termed as TI in the paper), and LoRA [25]. Since there are currently no available studies that focus on locating 'high-quality' samples for training, we apply random sampling (RAND), and CLIP-score-based sampling (CLIP) for each baseline in our active learning setting. Each experiment starts with 20 randomly sampled instances, and we conducted 5 cycles of data accumulation in which we selected 20 'good' samples to add to the training pool. We train 20 epochs for all combinations of adaption techniques and sampling strategies in each active learning cycle, with a batch size of 1. Furthermore, we generate 100 images for each concept for evaluation. We use the commonly adopted Stable Diffusion 2.0 pre-trained on LAION-5B [51] following Rombach's work [45].

Evaluation Metrics. We evaluate the quality of our generated images with the widely used CLIP score [20] and the recently proposed CMMD score [27], which quantify the model performance in two aspects. Specifically, the CLIP score measures how generated images match the given text, which is expected to be as high as possible. Meanwhile, the CMMD score evaluates the discrepancy between generated images with the real ones, indicating better generative results with lower values.

5.1 Single-concept Generation Results

To evaluate the performance of CATOD on OOD concepts, we test it on all 25 target concepts one by one separately and report the average performance of 5 concepts within each category in Table 1. We show the superior results of our CATOD with respect to both qualitative and quantitative comparisons.

Qualitative Comparisons. We qualitatively compare CATOD with other sampling strategies according to their generated images based on LoRA [25]. With random sampling (RAND), we observe that the generative results only partially learned some photographic attributes like color and texture,

Ground Truth

RAND

CLIP

CATOD

but failed to make the objects have the correct appearance and shape. CLIP-based sampling (CLIP) somehow produces the corrected shape of the concept but still fails to capture the necessary details for describing the object. In comparison, our proposed CATOD successfully matches all the photographic attributes, while also guaranteeing the image aesthetics.

To further look at how CATOD learns the photographic attributes that precisely match the concept, we show samples generated from different cycles in Fig. 7. We can see that some attributes like color, and texture are already learned in cycle 1, but the shape of the object does not match the ground-truth ones. From cycle 1 to 3, CATOD shows a clear shape modification, making the objects more like the real ones. From cycles 3 to 5, an iterative refinement on more photographic details like light, contrast, and other minor modifications (like the beak for penguins and antenna for axolotls) is shown in generative results, making them hard to distinguish from the ground-truth ones even with a careful look. At cycle 5 and later cycles, the image quality stabilizes and we can hardly see enhancement apart from image diversity. To conclude, we can see an explicit attribute matching process from easy ones to the finer ones within CATOD, showing the importance and effectiveness of iterative training.

Quantitative Comparisons. Table 1 reports CLIP scores [20] and CMMD scores [27] from each strategy with models trained on 25 different OOD concepts and evaluated through the generative results. Specifically, we evaluate the performance of CATOD on each concept and average the results within each category in Table 1. We have the following conclusions: (1) the average performance of our strategies outperforms all compared methods by a 0.56 ∼ 11.10 CLIP score, justifying the effectiveness of locating aesthetic and clean samples over text-image matching. (2) CATOD also brings a 0.12 ∼ 0.54 CMMD decrease on various frameworks and concepts, indicating better image alignment with proper sampling guided by CATOD. To summarize, both image-matching and textmatching scores exhibit better results compared to the baselines, suggesting that our proposed CATOD significantly outperforms other strategies, which is consistent with the qualitative results in Fig. 4.

Ground Truth

(Unseen Concept)

Cycle 1

Cycle 2

Cycle 3

Cycle 4

Cycle 5

Cycle 6

Cycle 7

5.2 Multi-concept adaption Results.

To further investigate whether CATOD could adapt multiple concepts simultaneously, we group the 25 concepts by category and train the adaptation model on each category. To be specific, we compare our baselines using a generated set consisting of 500 images (100 images per concept) and exhibit our results in Table 2. Following the setting of single-concept adaption, we conduct 10 cycles of data accumulation and get 200 samples at last, since multi-concept adaption requires more data. We still observe a notable performance gain with a 1.56 ∼ 5.07 CLIP score increase and a 0.12 ∼ 0.86 CMMDscore decrease compared to baselines. These results verify that our CATOD still achieves better performance on multi-concept adaption.

5.3 Ablation Studies

We verify the efficacy of all components in our proposed CATOD in Table 3 and 4, including the aesthetical scoring module and the concept-matching mechanism within the weighted scoring system.

W/OAesthetic Score. First, we validate the effectiveness of CATOD by removing the aesthetic scores. A significant decrease in performance (up to 8.55) on the CLIP score can be observed in Table 3. We attribute this decline to the fact that matching-based metrics prioritize image representations over the given concept, leading to a deterioration in image-text matching.

The Type of Aesthetic Scores. To further investigate the impact of aesthetic scores, we have also applied different types of aesthetics with CATOD in Table 4. We observed that recent state-of-theart aesthetic evaluations did not improve OOD adaption and even led to minor performance loss. This could be attributed to the fact that these models were originally designed for general aesthetic assessments, whereas our aesthetic scorer is highly personalized towards specific OOD concepts.

W/O Concept-Matching Score. After removing the concept-matching scorer, we observed that the adaptor tends to perform better compared to just removing the aesthetic scorers. This might be due to the aesthetic scorer being designed based on CLIP backbones, showing some consistency with the CLIP score. However, it still demonstrates limitations based on the image-matching score CMMD. While these aesthetic qualities partly describe the clarity and accuracy of the object, they do not adequately focus on the image representation space.

The Weighted Score. We have confirmed the effectiveness of balancing two scores by simply adding them together to guide CATOD (Table 3, Line 5). We observed that this resulted in consistent performance loss for both the CLIP score (up to 1.65) and the CMMD score (up to 0.22). This suggests that both text-image matching and image-to-image matching are affected by the trade-off between aesthetics and concept matching, further emphasizing the importance of these two scores.

6 Related work

Personalized Text-to-image Synthesis with Adaptors. The task of text-to-image generation involves creating specific images based on text descriptions [3, 73, 79, 4], and has achieved impressive performance with state-of-the-art diffusion models [41, 45, 39]. Therefore, adapting large-scale text-to-image models to a specific concept while also preserving this amazing performance, i.e. personalization [7], has become another recent research interest. But this is often difficult since re-training a model with an expanded dataset for each new concept is prohibitively expensive while fine-tuning the whole model [12, 32] or transformation modules [83, 20, 60] on few examples typically leads to some rate of forgetting [30]. Therefore, adaptors such as Textual Inversion [18], DreamBooth [47, 48, 5, 58], LoRA [25, 66, 86, 76], along with some other works [70, 19] derived from them, have become more commonly adopted. Typically, they focus on a small but crucial part of the model or extra networks inserted into underlying models, thus more computationally-efficient, while also preserving the efficacy of the underlying models with lower computational costs. For example, textual inversion (TI) [18] represents the newly-given concept with pseudo word [42] and remapping it to another carefully trained embedding in the text-encoding space, guided by few images. Despite their computational efficiency, these approaches are still facing difficulties dealing with out-of-distribution concepts as we observe in Section 2.

Active Learning and Selection. Active Learning is a machine learning paradigm that involves actively selecting the most suitable data for training models from external data sources [44, 63]. The most crucial part of active learning is the strategy to locate the optimal data batch. Current studies can be roughly categorized as follows: (a) Score-based methods that prefer the samples with the highest information scores [36, 69, 10]; (b) Representation-based methods searching for the samples that are the most representative of the underlying data distribution [53, 1, 62]. The Active Selection paradigm within Active Learning serves as an efficient and powerful dataset curation tool (i.e. adaption ), leading to numerous studies adopting this paradigm from a wide range of subjects [68, 24, 8]. Meanwhile, due to the effectiveness of Active Learning in selecting the most suitable training samples, recent studies have utilized similar sampling strategies to address challenges associated with long-tailed distributions [56, 57] and noisy data [72]. Given that Out-Of-Distribution (OOD) concepts in Latent Diffusion Models (LDMs) often involve unseen or long-tailed concept tokens [79], this motivates us to leverage Active Learning for selecting samples that are well-suited for training adaptors.

In this work, we focus on scored-based strategies in two-fold: aesthetics and concept-matching . Image Aesthetics Assessment (IAA) aims at evaluating image aesthetics computationally and automatically [9], while automatically assessing image aesthetics is useful for many applications [35, 29, 14]. To take a step further, personalized image aesthetics assessment (PIAA) [43, 85, 75] was proposed to capture unique aesthetic preferences, consistent with our goal to adjust our paradigms accordingly different concepts. At the same time, "Concept-matching" is adopted from the field of image retrieval, in which we search for relevant images in an image gallery by analyzing the visual content (e.g., objects, colors, textures, shapes etc .), given a query image [61, 31]. To design a paradigm that automatically meets the harsh requirements for adaptor training, we consider both two factors.

7 Conclusion

We propose CATOD, an enhanced, data-efficient, and practically useful version of the OOD concept adaptation for AIGC. This method is encapsulated in an active-learned paradigm with carefully designed acquisitional scoring mechanisms. CATOD significantly outperforms the prior approaches in many (if not most) aspects including generation quality, concept matches, technological robustness, data efficiency, etc. In the future, we hope to ship CATOD to the open-source community so as to absorb more OOD concepts that were originally uncovered.

References

Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process on bidirectional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41(5):1213-1226, 2018.

Razan Bayoumi, Marco Alfonse, and Abdel-Badeeh M Salem. Text-to-image synthesis: A comparative study. In Digital Transformation Technology: Proceedings of ITAF 2020 , pages 229-251. Springer, 2021.

Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279 , 2024.

Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03374 , 2023.

Edwin K. P. Chong. Well-conditioned linear minimum mean square error estimation. IEEE Control. Syst. Lett. , 6:2431-2436, 2022.

Niv Cohen, Rinon Gal, Eli A Meirom, Gal Chechik, and Yuval Atzmon. 'this is my unicorn, fluffy': Personalizing frozen vision-language representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX , pages 558-577. Springer, 2022.

Wupeng Deng, Quan Liu, Feifan Zhao, Duc Truong Pham, Jiwei Hu, Yongjing Wang, and Zude Zhou. Learning by doing: A dual-loop implementation architecture of deep active learning and human-machine collaboration for smart robot vision. Robotics and Computer-Integrated Manufacturing , 86:102673, 2024.

Yubin Deng, Chen Change Loy, and Xiaoou Tang. Image aesthetic assessment: An experimental survey. IEEE Signal Processing Magazine , 34(4):80-106, 2017.

Francesco Di Fiore, Michela Nardelli, and Laura Mainini. Active learning and bayesian optimization: a unified perspective to learn with a goal. Archives of Computational Methods in Engineering , pages 1-29, 2024.

Mario Díaz, Peter Kairouz, and Lalitha Sankar. Lower bounds for the minimum mean-square error via neural network-based estimation. CoRR , abs/2108.12851, 2021.

Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the clip model. arXiv preprint arXiv:2207.09248 , 2022.

Alex Dytso, H Vincent Poor, Ronit Bustin, and Shlomo Shamai. On the structure of the least favorable prior distributions. In 2018 IEEE International Symposium on Information Theory (ISIT) , pages 1081-1085. IEEE, 2018.

Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3677-3686, 2020.

Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11665 , 2021.

Li Fe-Fei et al. A bayesian approach to unsupervised one-shot learning of object categories. In proceedings ninth IEEE international conference on computer vision , pages 1134-1141. IEEE, 2003.

Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, and Bharath K Sriperumbudur. Characteristic kernels on groups and semigroups. Advances in neural information processing systems , 21, 2008.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel CohenOr. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG) , 42(4):1-13, 2023.

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2110.04544 , 2021.

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307.12980 , 2023.

Dongning Guo, Shlomo Shamai, and Sergio Verdú. Estimation of non-gaussian random variables in gaussian noise: Properties of the MMSE. In Frank R. Kschischang and En-Hui Yang, editors, 2008 IEEE International Symposium on Information Theory, ISIT 2008, Toronto, ON, Canada, July 6-11, 2008 , pages 1083-1087. IEEE, 2008.

Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. Rethinking image aesthetics assessment: Models, datasets and benchmarks. In Proceeding of the Thirty-First International Joint Conference on Artificial Intelligence , 2022.

Aral Hekimoglu, Michael Schmidt, and Alvaro Marcos-Ramiro. Monocular 3d object detection with lidar guided semi supervised active learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2346-2355, 2024.

Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2021.

Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13269 , 2023.

Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. arXiv preprint arXiv:2401.09603 , 2023.

Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. In The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010 , pages 3304-3311. IEEE Computer Society, 2010.

Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14 , pages 662-679. Springer, 2016.

Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Finetuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054 , 2022.

Michael S Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) , 2(1):1-19, 2006.

Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu. Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 5441-5454, 2022.

Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, and Weisi Lin. Personality-assisted multi-task learning for generic and personalized image aesthetics assessment. IEEE Transactions on Image Processing , 29:3898-3910, 2020.

Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. Advances in Neural Information Processing Systems , 32, 2019.

Yiwen Luo and Xiaoou Tang. Photo and video quality evaluation: Focusing on the subject. In Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10 , pages 386-399. Springer, 2008.

Xiaoyi Mai, Salman Avestimehr, Antonio Ortega, and Mahdi Soltanolkotabi. On the effectiveness of active learning by uncertainty sampling in classification of high-dimensional gaussian mixture data. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4238-4242. IEEE, 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Michael A Nielsen. Neural networks and deep learning , volume 25. Determination press San Francisco, CA, USA, 2015.

Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Natalie Rathvon. Early Reading Assessment. A Practitioner's Handbook. ERIC, 2004.

Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David J Foran. Personalized image aesthetics. In Proceedings of the IEEE international conference on computer vision , pages 638-647, 2017.

Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR) , 54(9):1-40, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18 , pages 234-241. Springer, 2015.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949 , 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.11487 , 2022.

Hitesh Sapkota and Qi Yu. Balancing bias and variance for active weakly supervised learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1536-1546, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Silvio Savarese Sener et al. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

Burr Settles. Active learning literature survey. 2009.

Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui, André Araújo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023 , pages 1100211012. IEEE, 2023.

Jiang-Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learning? Advances in Neural Information Processing Systems , 36:75669-75687, 2023.

Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin-Yan Han, and Yu-Feng Li. Longtail learning with foundation model: Heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning , pages 45014-45039, 2024.

Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411 , 2023.

Josef Sivic and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In Computer Vision, IEEE International Conference on , volume 3, pages 1470-1470. IEEE Computer Society, 2003.

Gabriel Skantze and Bram Willemsen. Collie: Continual learning of language grounding from language-image embeddings. Journal of Artificial Intelligence Research , 74:1201-1223, 2022.

Arnold WM Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. Content-based image retrieval at the end of the early years. IEEE Transactions on pattern analysis and machine intelligence , 22(12):1349-1380, 2000.

Qun Sui and Sujit K Ghosh. Similarity-based active learning methods. Expert Systems with Applications , 251:123849, 2024.

Alaa Tharwat and Wolfram Schenck. A survey on active learning: state-of-the-art, practical challenges and research directions. Mathematics , 11(4):820, 2023.

Weijie Tu, Weijian Deng, Tom Gedeon, and Liang Zheng. A bag-of-prototypes representation for dataset-level applications. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 2881-2892. IEEE, 2023.

Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023 , pages 1921-1930. IEEE, 2023.

Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameterefficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pages 3266-3279, 2023.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8769-8778, 2018.

Jinsheng Wang, Guoji Xu, Peng Yuan, Yongle Li, and Ahsan Kareem. An efficient and versatile kriging-based active learning method for structural reliability analysis. Reliability Engineering &System Safety , 241:109670, 2024.

Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Cheng-Zhong Xu, and Min Xu. Boosting active learning via improving test performance. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 8566-8574, 2022.

Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1594315953, 2023.

Xiaoping Wu, Chi Zhan, Yu-Kun Lai, Ming-Ming Cheng, and Jufeng Yang. Ip102: A large-scale benchmark dataset for insect pest recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8787-8796, 2019.

Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang, and Yu-Feng Li. Ngc: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 62-71, 2021.

Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. RAPHAEL: text-to-image generation via large mixture of diffusion paths. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

Yuzhe Yang, Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo. Personalized image aesthetics assessment with rich attributes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19861-19869, 2022.

Shin-Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. arXiv preprint arXiv:2309.14859 , 2023.

Ran Yi, Haoyuan Tian, Zhihao Gu, Yu-Kun Lai, and Paul L Rosin. Towards artistic image aesthetics assessment: a large-scale dataset and a new method. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22388-22397, 2023.

Mariia Zameshina, Olivier Teytaud, and Laurent Najman. Diverse diffusion: Enhancing image diversity in text-to-image generation. arXiv preprint arXiv:2310.12583 , 2023.

Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909 , 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Lin Zhao, Meimei Shang, Fei Gao, Rongsheng Li, Fei Huang, and Jun Yu. Representation learning of image composition for aesthetic prediction. Computer Vision and Image Understanding , 199:103024, 2020.

Yan Zhao, Lei Wang, Luping Zhou, Yinghuan Shi, and Yang Gao. Modelling diffusion process by deep neural networks for image retrieval. In BMVC , page 161, 2018.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision , 130(9):2337-2348, 2022.

Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. Personalized image aesthetics assessment via meta-learning with bilevel gradient optimization. IEEE Transactions on Cybernetics , 52(3):1798-1811, 2020.

Hancheng Zhu, Yong Zhou, Leida Li, Yaqian Li, and Yandong Guo. Learning personalized image aesthetics from subjective and objective attributes. IEEE Transactions on Multimedia , 2021.

Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Deltalora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411 , 2023.

A Theory

This section provides a complete derivation for the analysis given in Section 4. In brief, we first link L LDM to a minimum mean square error (MMSE) term in Theorem A.2 (i.e. Theorem 3.2 within the main context), then decompose the MMSE term to see how we minimize MMSE in different aspects in Theorem A.3 (i.e. Theorem 3.3 within the main context) in the main context. For convenience, we reclaim some of the formulations at the beginning of this section:

Definition A.1. The minimum mean square error (MMSE) of estimating an input random vector ̂ X ∈ R n from an observation/output X ∈ R k is defined as

in which M ( R n ) denotes the space consisting of all measurable functions on R n .

Based on Eq. (1), we can adapt the LDM to an arbitrary concept y with a corresponding image set X that describes this concept. Typical adaptations are mostly based on fine-tuning all the parameters θ , which is quite costly and requires a lot of data. However, recent studies have found that training only a small part of the model [18, 47] or inserting extra networks [25, 26] could attain the same performance, which largely alleviates the computational burden with far fewer samples need for training. We call this part of parameters or networks as ' adaptors ' and denote them by A . Then, the fine-tuning process of the adaptors , also named adaption , can be formulated as follows:

Therefore, v be a conditioning vector corresponding to some given text y , the LDM loss is:

Following the definition of LDM loss, we continue to link LDM loss to the MMSE term in the theorem as follows (in which we omit the parameter y since it is independent of other parameters):

Theorem A.2. Let L LDM be the LDM loss following Eq. (13) , and the image space lies within R n . Then there always exists an ideal random vector ̂ X G ∈ R n and a condition vector v ∗ within the text embedding space for LDM, such that

Proof. By denoting an ideal generative distribution with ̂ X G , which is produced by the generative model with a minimized LDM loss, we continue our proof. Set f A ( X T ) = E x ∼ X T [ x, A ] , we can see that f A is also a functional controlled by the adaptor A . Following the definition of MMSE and a fixed X T , we set A ∗ as:

With this A ∗ , and the Universal Approximation Theorem [38] for all deep learning models, f A ∗ becomes the functional needed to quantify the MMSE term following its definition. Therefore, with proper X G , A ∗ , we have

thus completing the proof of this theorem.

Theorem A.3. (Pythagorean Theorem for MMSE [13].) Following Theorem A.2, by setting f to the generative model g A ∗ with an ideal adaptor A ∗ containing sufficient OOD concept information, rewrite g A ( X T ) to X G | X T , and the minimum mean square error (MMSE) in Eq. (11) can be decomposed into two terms:

Proof. This equation can be further decomposed into two terms:

Since ̂ X G -X G represents the variance for generative results and is orthogonal to X T , the first term is 0, making it sufficient to consider only the second term. To prove that the second term is also 0, we first prove the orthogonality property, i,e.

for any function ϕ .

Therefore, we have the orthogonal property in Eq. (16).

Now we continue our proof for the main theorem. First, we can decompose Eq. (15) as follows:

Since the gap between pure generative result guided by the training set X T and an ideal text embedding is purely influenced by the training set X T , g A ∗ ( X T ) -E [ X G | X T ] can also be regarded as a functional over X T , making the last term equal to 0 according to the orthogonal property in Eq. (16). At this step, we obtain the result in Eq. (15).

B More Details About CATOD.

In this section, we provide more details about our proposed Controllable Adaptor Towards Out-ofDistribution Concepts (CATOD).

B.1 An Overall Paradigm

Initially, we have a large data pool D pool related to the concept given with different quality and begin with an initial randomly-sampled training dataset X (0) T . X T might contain distorted images or even mismatch the given concept, which is common in publicly available datasets. Following the typical paradigm of Active Learning, we train all the models we use on the training set X T . Then, we select a batch of data B from the data pool D pool according to the acquisition functions induced from models. Finally, we move this selected batch of data B to X T , go back to the model training step to re-train or adapt the models, and repeat this cycle until X T reaches its maximum capacity or the quality of models cannot be further enhanced by adding new data. An overall Algorithm for a cycle of CATOD is provided in Alg. 1.

Algorithm 1 An active selection cycle for CATOD.

Input : Text-to-image model g A ( · ) that can be integrated with adaptor A , real-world data pool D pool , training data pool X T , budget b , pretrained aesthetic scorer S aes ( · ) , concept-matching scorer S con ( · ) , A R

̸

adaptor , learning rate group . Output : Updated training pool X T , updated adaptor A . 1: Fine-tune the aesthetic scorer S aes ( · , θ ) on X T following Sec.B.3; 2: Calculate the aesthetic score γ aes ( A ) = 1 | g A ( X T ) | ∑ x g ∈ g A ( X T ) S aes ( x g ) and concept-matching score γ con ( A ) = 1 | g A ( X T ) | ∑ x g ∈ g A ( X T ) S con ( x g ) for A ; 3: for x ∈ D pool do 4: Calculate the integrated score S ( x ) (Eq.(7)); 5: end for 6: B ← {Top-b samples within D pool according to S }; 7: s ← γ ( A ) (Eq.(6)); 8: r ← the largest value within R ; 9: while r = min( R ) do 10: A ′ ← A -r ∇ A ∑ x ∈ X T ∪B L LDM ( x, A ) ; 11: Generate a small-scale X G with g ( · ) conditioned on A ′ ; 12: Calculate score γ ( A ) for A via Eq.(6)) and X G ; 13: if γ ( A ) > s then 14: A ← A ′ ; 15: s ← γ ( A ) ; 16: else 17: r ← a smaller value than r within R 18: end if 19: end while 20: X T ← X T ∪ B

We use a weighted-scoring system as the acquisition module that evaluates the quality of images and adaptors in our proposed CATOD framework. In traditional active learning or data selection, the acquisition module is only conducted on samples. However, we observed that an increase in the number of samples does not necessarily contribute to a performance boost like in a traditional active learning setting (which we will verify in Appendix C). Therefore, we propose this weighted scoring system to help schedule adaptor training, which helps locate the best version among them and select samples according to their quality.

Our Controllable Adaptor Towards Out-of-Distribution Concepts framework can be divided into 4 steps: (1) Train the scoring system on the current training set X T . (2) Train the adaptor on the training set X T and schedule the training accordingly to the scoring system; (3) Calculate the acquisition score accordingly to the scoring system and the best adaptor for samples in dataset D pool ; (4) Select top-K samples from D pool and move them to train set X T .

B.2 An Implementation of MMD score.

To give out an quantifiable and unbiased estimation of Eq.(2), we sample two sets of vectors X = { x 1 , x 2 , . . . , x m } from P and G = { g 1 , g 2 , . . . , g n } from Q , an estimation can be give by:

̸

̸

B.3 Aesthetic Scorer

Following the description of the aesthetic scorer in Section 3.2, we explain how to personalize it here. Since our primary goal is to make the model distinguish low-quality/high-quality examples for OOD concepts, we design an automatic procedure to assign aesthetic scores to our training set X T at each cycle, based on which we attain an aesthetic training set X aes T . In this work, the aesthetics training comprises three parts: (1) The original training set X T ; (2) Generated set D TG without assistance

High Aesthetic

High Concept-Matching

High Quality

Ideal Image Manifold

Latent

Variable

Aesthetic

Samples

Concept-Matching

Samples

Fixed Schedule (no trade-off)

Dynamic Schedule

from any adaptors; (3) Randomly sampled set D SI from similar but irrelevant categories. Since we focus on the concepts that the text-to-image model can hardly recognize and visualize, D TG contains lots of plausible, disrupted, or even irrelevant samples, which we assign a zero score. This helps the aesthetic model understand that originally generated samples are low-quality samples. In contrast, D SI somehow describes some attributes for the concept but still does not match the concept, to which we assign the average score of 5.0. We employ the normal regression loss formulated as follows:

where p denotes score predicted by the original scorer, when ˆ p denotes the score we assign, and λ is set to 1.0 in our experiments. Following this paradigm, we train an aesthetic scorer for each category of concepts, and the score personalized for each category from D pool can be predicted accordingly.

After selecting top-K data according to this score based on adaptor quality, we assign a lowered aesthetic score to them:

where p x denotes the score given by a generic aesthetic scorer, total represents the number of total active learning cycles, when t current indicates the current cycle. Note that the newly selected images typically have lower quality than those from earlier cycles, we assign progressively lowered scores to them as the learning cycle proceeds.

B.4 Adaptor Evaluation and Schedules

Following the content in the second part 'The Active Schedule for Training Adaptors' in Sec. 3.3, we describe our schedule in more detail. In training CATOD, we use this indicator as a signaling proxy throughout the training process. As the number of training samples increases, the quality of selected samples at later cycles might be lower, and a fixed schedule may introduce some unnecessary details within these images (such as watermarks, borders, disturbing objects, etc.). However, these subsequent samples with relatively lower quality do contain some good photographic attributes that can help the embedding evolve. Actually, the potential problems can be alleviated by carefully-designed schedules. An intuitive illustration for this schedule is given in Fig. 6.

We first train the adaptor for 5 epochs, and save an adaptor version A i per 5 epochs, together with its corresponding generated images, constituting a sub-cycle. When this sub-cycle is completed, we filter out the adaptor with the highest quality based on the score γ ( A ) (Eq.(6)). If the selected adaptor has the most training epochs across all versions, we keep the learning rate and conduct the next sub-cycle, otherwise, we will reduce the learning rate. Note that if the adaptor quality of this cycle is not as good as that of the previous cycle (because of the quality degradation caused by introducing some unnecessary details), the initial learning rate will be reduced, which helps avoid

introducing disturbing elements led by some samples with insufficient quality. The sub-cycles will be continuously conducted until the minimum learning rate is reached or the adaptor quality no longer increases. Our learning rates include 5 × 10 -4 , 2 . 5 × 10 -4 , 7 . 5 × 10 -5 , 5 × 10 -5 , 2 . 5 × 10 -5 .

C Experiments

C.1 How to Automatically Locate OOD Concepts

Since the publicly available dataset, i.e., ImageNet, iNaturalist 2018 [67], IP102 [71], which we adopt in our research contains 9,244 concepts with 14,883,455 images in total, it is infeasible to manually test and decide what concept is the out-of-distribution concept one by one. Therefore, our procedure for figuring out is two-fold: (1) automatically quantifying the distribution drift of the given concept from what the text-to-image model (Stable Diffusion 2.0, a.k.a SD 2.0) learned with

the CMMD metric (thanks to the recent work proposed by Google [27]); (2) Manually verifying whether the text-to-image model could generate the right content, according to the rank given by the CMMD metric above. Specifically, in the first step, we randomly sample 10 image data for each concept within the datasets and then generate 10 images with the model with the text prompt 'A photo of S ∗ ', in which S ∗ denotes the concept name. Then, we calculate the CMMD discrepancy between these two data batches and record them. The CMMD scores of some concepts (including both Out-of-Distribution concepts in the first 5 rows and In-Distribution in the last 5 rows) are listed in Table 5.

Empirically, we observed that SD 2.0 is unable to synthesize the concepts with a CMMD metric above 3.5. Therefore, we pick 25 OOD concepts accordingly divide them into 5 categories, and make our dataset as follows:

Insect : Zopherinae, Antlion, Lycorma Delicatula, Parasitic Wasps, Xylotrechus;

Lizard : Axolotl, Frilled Lizard, Mediterranean House Gecko, Oedura, Opluridae;

Penguin : Emperor Penguin Chick, Gentoo Penguin, King Penguin Chick, Rock Hopper Penguin, Royal Penguin;

Sea Fish : Crampfish, Dragonfish, Garfish, Tigerfish, Tuna;

Snake : Ahaetulla Nasutar, Aipysurus Laevis, Indian Cobra, Pelamis Platurus, Sidewinder.

Actually, the dataset we collect is not class-balanced, so the number of samples we can collect varies across different concepts. The statistics of each concept are shown in Table 6. For the concepts with a D pool less than 1,000 images, we also collect some of them from image host websites including Dreamstime, Flicker, istockphoto, pinterest, and shutterstock. The dataset will be released as soon as the code is released.

C.2 Experimental Settings

To verify the effectiveness and extensibility of our proposed CATOD framework, we conduct experiments on 25 different OOD concepts that the large-scale text-to-image (i.e. Stable Diffusion 2.0 [45]) implement different versions of CATOD. This section explains some important implementation details for the dataset, the aesthetic scorer, the concept-matching scorer we use, and how we design the learning paradigm.

The Learning Rate Schedule. As shown in Algorithm 1, we have a learning rate group R, which includes 5 learning rates: 5 × 10 -4 , 2 . 5 × 10 -4 , 7 . 5 × 10 -5 , 5 × 10 -5 , 2 . 5 × 10 -5 . Following the indicator γ ( A ) in Eq. (6), we calculate this value after every epoch. When this indicator falls below the previous evaluation, we reduce the learning rate. If the learning rate cannot be lowered further, we conclude that the model has converged and the training is complete. Note that we choose to train 20 epochs in all experiments as claimed in the main context, it is possible that the training is early stopped before reaching epoch 20.

Aesthetic Scorers. To offer the basic knowledge for aesthetic evaluation, we use pre-trained generic models, including ReLIC [81], TANet [23], SD-Chad Scorer, and VLAD [59]. Note that the scores given by the generic aesthetic scoring model tend to lie in the majority score range (5 to 6) in our dataset, we ought to personalize this model accordingly to our training set, which we refer to as PIAA [84]. Since PIAA is a typical small sample problem, we adopt similar experimental settings and evaluation criteria by referring to Few-Shot Learning [16] and a previous PIAA research work: PA-IAA [33].

Concept-Matching Scorers. Since the first step of general image retrieval is feature extraction, we use the CLIP ViT-L/14 encoder, which is a commonly used and reliable feature extractor [2, 82, 34].

C.3 About Compositional Results over Multiple-Concepts.

To further validate the effectiveness of CATOD, we also present compositional results based on the LoRA adaptors we obtained, as illustrated in Figure 7. We compose different types of concepts, including (i) background concepts such as brick wall and grassland for Emperor Penguin Chicks,

Emperor

Penguin

Chick

Frilled

Lizard

Ground

Truth

+ on desk

+ in the beach

+ with a dog

+ with an

Alligator Lizard

+ with an

Opluridae

Ground

Truth

+ against

brick wall

+ in the

grassland

+ with a dog

+ with an

+ with an King

and desk and beach for Frilled Lizard; (ii) in-distribution and common concepts, like the dog; (iii) in-distribution and similar concepts, like Adelie penguin for Emperor Penguin Chicks, and Alligator Lizard for Frilled Lizard; (iii) out-of-distribution concepts from other adaptors, like King Penguin Chicks for Emperor Penguin Chicks and Opluridae for Frilled Lizard. Our observations reveal that the synthetic results on out-of-distribution concepts with CATOD can seamlessly integrate different background elements, even if they were not present in real-world images (columns 1 to 2). For other in-distribution concepts like dogs, LDMs tend to represent a specific species with similar color and texture, while different concepts remain highly distinguishable within one image (columns 3 to 4). For out-of-distribution concepts, we also note that the creatures are depicted correctly, but may exhibit some confused visual details that negatively impact the aesthetics of the image (column 5).

C.4 More Analysis.

10

2.5e-4

7.5e-5

5e-5

5-992

20%

6

4

20%

20%

20%

14%

14%

14%

28%

20%

20%

20%

33%

22%

22%

50%

16%

20%

20%

28%

20%

11%

11%

16%

16%

0

1

No. Cycle

2

3

4

aesthetic

tag-matching

indicator

10

6

8765

4

3

Q

3

6

No. Cycle

6

12

15

18

CATOD is more reliable on schedules. Figure 8(a),(b) show a comparison of how our schedules adjust as the number of cycles increases on RAND(scheduled) and CATOD. Both of them show the tendency that training epochs with larger learning rates decrease when those with small learning rates

TI-RAND(Fixed)

TI-RAND

CATOD(Fixed)

CATOD

06

80

70

50

40

30

20

10

0

6

9

12

15

18

No.Cycle

Shape 1

Shape 2

Shape 3

Ground

Truth

Generated

Images

increase, meaning that the training schedule focuses more on fine-tuning at later cycles. Comparing CATOD with RAND, we can see that the epochs with the maximal learning rate already diminish at cycle 4 on CATOD while RAND still needs large learning rates, indicating that CATOD are better at recognizing the given concept and tends to be more stable.

CATOD achieves a good score earlier than other methods. Figure 8(c),(d) shows a comparison of how the aesthetic/concept-matching/comprehensive scores change as the cycle proceeds on RAND and CATOD. We can see that CATOD already achieves a higher value on all scores and cannot be further boosted with 120 training samples on cycle 12 when the performance for RAND still fluctuates. This phenomenon illustrates that our active selection strategy makes our embedding be trained on more high-quality data.

Furthermore, we also notice that aesthetic/concept-matching scores fluctuate at later cycles when the comprehensive score is enhanced continuously, which exactly corresponds to our proposed dual scoring system that balances the importance of these two factors when ensuring the quality of the adaptor never declines.

The performance of CATOD on different initial training data pool. We also ablate CATOD over the initial training set size/quality. In detail, we retest CATOD with different numbers of initial samples (10,20,50) and different quality, i.e., high-quality (HQ) and random sampling (RAND), with 100 images sampled at last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results are shown in Table 7:

From these results, we can draw the following conclusions:

Initial batch size has a more significant impact on randomly initialized samples than on high-quality samples. Specifically, the performance change with high-quality samples is at most 1.40 in the CLIP score and 0.10 in the CMMD score concerning different initial numbers of training data. In contrast, the performance change on low-quality is up to 7.65 in the CLIP score and 0.45 in the CMMD score.

The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.

C.5 Further Extension: Concept with multiple shapes

Recall that Stable Diffusion 2.0 fails to generate emperor penguin chicks, which accounts for the fact that adult emperor penguin chicks and their chicks have different appearances. We raise another question: can we train an adaptor with a concept that has different shapes? Inspired by this, we also test our proposed CATOD on the concept 'lycorma delicatula', which is a kind of pest with 3 different forms in its life-cycle, and surprisingly find that with 300 training samples, the best version of adaptors successfully produces all these 3 shapes, as shown in Figure 9(b). These results further validate the effectiveness of our framework, even in the presence of appearance ambiguity.

C.6 About Experimental Results on other Architectures

Our experiments in the main content are all conducted on Stable Diffusion 2.0 (SD 2.0). To further validate the superiority of our proposed CATOD, we conduct additional experiments on other architectures (based on LoRA), including Stable Diffusion 1.5 (SD 1.5) and Stable Diffusion XL (SDXL). The results are shown in Table 9.

This table shows that our proposed CATOD is also compatible with different architectures with notable performance gain compared to the baselines.

Selection

Generation

LoRA + CLIP

Axolotl

LoRA + CATOD

LoRA + CLIP

Emperor

Penguin

Chick

LoRA + CATOD

C.7 About the diversity

CATOD maintains the diversity to produce OOD concepts with different angles or poses in generative results. To validate the diversity of our generative results, we first provide a quantitative evaluation with LPIPS [80, 78]. The results are shown in Table 9. In this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity. Since the training set contains samples with different angles in CATOD, it is also easy to produce objects with different angles, and we show the examples on concepts 'Axolotl' and 'Emperor Penguin Chick' in Figure 9. We can observe that training samples with different angles selected by CATOD also lead to diverse angle in their generative results.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have clearly summarized our contribution in the abstract and in the last paragraph of the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: We believe that the limitation should not be regarded as an integral part of this paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have provided theoretical proof of our theory in Sec.4 in Appendix. A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided our code within the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: This paper does not provide the dataset regarding the risk of anonymity, despite that the dataset is just a collection of publicly available datasets.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All experimental details are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Our experiments do not have error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This is reported in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our paper has no risk regarding ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This has been briefly discussed in our conclusion section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: There is no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: MIT License has been added already.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have a readme document for our codes.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No crowdsourcing experiments were conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No crowdsourcing experiments were conducted.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CATOD, a framework designed to enhance the adaptation of text-to-image models for out-of-distribution (OOD) concepts. It addresses the issue of low-quality training data by employing an active learning approach that iteratively improves the training set. The framework utilizes a scoring system comprising aesthetic and concept-matching scores to guide data selection and training. This scoring system dynamically determines the relative weight of the aesthetic score and content matching score to prioritize samples for selecting the training set. The authors demonstrate significant performance improvements using CATOD, achieving notable improvements on standard metrics.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n2. The research problem of out-of-distribution (OOD) on text-to-image models is essential, and the motivation is quite clear.\n3. The approach is innovative, which combines active learning with a novel scoring system to enhance model adaptation.\n4. The paper has very adequate quantitative experimental results and show clear improvements over baseline methods.\n\nWEAKNESSES:\n1. The paper does not provide enough detail to clarify how the CMMD metric differs from other commonly-used metrics for image generation, such as Inception Score and FID. The authors should further explain this.\n2. Since your proposed CATOD is trained on few samples (100 for single-concept and 200 for multi-concept), the generation model may be biased toward specific samples. Reporting diversity metrics of the generated samples can help clarify this issue.\n\nQUESTIONS:\n1. I wonder if your proposed CATOD also outperforms other methods on previous metrics such as FID and the Inception Score.\n2. In the paper, random sampling and CLIP-score-based sampling are used as the comparison method, and finally 100-200 images are selected for training. Would it be better to fine-tune the model directly using all 1000 images?\n3. In addition to the three methods mentioned in the paper, are there other custom content generation methods that your proposed CATOD framework is still applicable to, such as LyCORIS[1] and ELITE[2]?\n\n[1] YEH, SHIH-YING, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. \"Navigating text-to-image customization: From lycoris fine-tuning to model evaluation.\" In *The Twelfth International Conference on Learning Representations*. 2023.\n\n[2] Wei, Yuxiang, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. \"Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.\" In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 15943-15953. 2023.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework called Controllable Adaptor Towards Out-of-Distribution (OOD) Concepts (CATOD). CATOD is designed to adapt text-to-image models to OOD concepts and generate high-quality images of those OOD concepts accordingly. The authors identified the challenge of accurately depicting OOD concepts due to low-quality training data. CATOD employs an active learning approach to iteratively accumulate high-quality training data and update the adaptor. The framework incorporates an aesthetic score and a concept-matching score to guide the accumulation of training data. Extensive experiments demonstrate significant improvements in both the CLIP score and the CMMD metric.\n\nSTRENGTHS:\n1. The presentation of the paper is clear and concise. The method is easy to understand and the delivery of the paper is smooth.\n2. The paper addresses a significant challenge in text-to-image generation by focusing on OOD concepts, and serves as the first work to address the OOD challenge from a data-centric perspective.\n3. The method is well-motivated and supported by theoretical insights into the importance of aesthetics and concept-matching scores.\n4. The experimental results demonstrate significant improvements over existing methods.\n\nWEAKNESSES:\n1. To support the claim in Line 189 that your selection \"guarantees the sample diversity\", the authors should report the diversity metrics of the generated samples. \n2. The paper provides ablation results on the design of weighted scoring strategies, which is valuable. I think more extensive ablation studies on the impact of initial data size, the type of text-to-image encoders, etc. can help signify the paper's contribution.\n3. To further investigate how the weighting strategy work in your proposed CATOD, it would better to list how the weights of aesthetic/concept-matching scores change as your learning cycle proceeds over different concepts.\n\nQUESTIONS:\n1. Following weakness 1, can you compare the diversity of the fine-tuned model under different strategies?\n2. It would be intriguing to investigate how initial data size impact the experiments, especially for the multiple concept scenario as you mentioned in Sec. 5.2 following weakness 2.\n3. For different concepts, which of the two scores gets high weightage (in equation 11) as the training progresses?\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the challenge of adapting text-to-image diffusion models to out-of-distribution (OOD) concepts. The authors introduce a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD), which employs an active learning paradigm to iteratively accumulate high-quality training data and optimize adaptor training. The framework features a dual scoring system that balances aesthetics and concept matching, ensuring improved quality and conciseness of the generated images.\n\nSTRENGTHS:\n1. This paper is easy to follow due to its clear writing, concise figures, and well-structured formulation of problem definitions, methods, and theories.\n2. The motivation of this paper is innovative, addressing the overlooked problem of adapting recent text-to-image (T2I) models to out-of-distribution concepts.\n3. The paper presents sufficient experimental results that effectively support its claims.\n\nWEAKNESSES:\n1.\tThe key contribution of this paper is Equation (6) and (7), which are claimed to be use as a signal to reduce the learning rate, stop the training in time, and select samples for the next cycle.  It seems the innovation is that the authors identify two relevant factors and utilize the strategy of active learning. Am I correct? \n2.\tAuthors use MMSE to reinterpret the loss function of the LDM and decompose it into two terms (Eq. 10). As the description in lines 213 to 220 of the paper, associating these two terms directly with aesthetic preservation and concept-matching score is rather far-fetched and vague. Kindly advise if I am wrong.\n3.\tIf the aesthetic and concept-matching scores are really that important, why just use those two scores to pick suitable samples for adaptation from the beginning? This would also avoid the later calculations.\n\nQUESTIONS:\n1. Is the CMMD shown in Figure 1 calculated using Real Images and before adapting images?\n2. Are the high-quality samples shown in Figure 2 generated by vanilla Textual Inversion, DreamBooth, and LoRA? Or are they already using the CATOD framework?\n3. Is “if r_k is the closet representation for r_k” in line 145 of the paper correct?\n4. Eq. 6 and the description in lines 173 to 174 of the paper confuse me. Shouldn’t the maximum value of \\gamma (A) be \\gamma_aes (A)=\\gamma_con (A)=10+40k, k=0...n, and the minimum value be \\gamma_aes (A)=30+40k, or \\gamma_con (A)=30+40k, k=0...n?\n5. Associating the two terms of Eq. 10 directly with aesthetic and concept-matching scores requires further theoretical explanation and empirical analysis.\n6. I noticed in the experiment section that the training period is about 20 epochs. Would this somehow be too less? If there were enough training cycles, we would get an overfitted LDM capable of generating images containing specific OOD concepts.\n7. In Algorithm 1, lines 9 to 19, it does not look like new training samples are being selected for each epoch.\n8. Does the learning rate set R determine the stopping of the training? How would you guarantee that the model has converged at this point? As question 5, a sufficiently overfitted LDM should be able to generate these OOD concepts.\n9. This paper tests the proposed CATOD on SD 2.0. Have you conducted experiments to evaluate its performance on other text-to-image models, such as SD 1.5 and SDXL? Including these results would help verify the versatility of your method.\n10. I am curious about how CATOD performs with different initial training set sizes (e.g., 10 and 50) and varying quality. Adding this as an ablation experiment would provide valuable insights into the robustness of your method.\n11. I am also interested in whether similar categories (e.g., other kinds of penguins) might be distorted after adaptation. Have you conducted any experiments to evaluate the degree of distortion in these cases?\n12. Since CMMD is a newly proposed metric by Google in CVPR 2024, it would be helpful to list the advantages of using CMMD compared to traditional metrics like Inception Score, FID, and MMD. This would help readers understand the rationale behind choosing CMMD.\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This work presents CATOD, a new method for dealing with the challenging scenario of adapting generative models to OOD concepts. In particular, CATOD leverages an active learning setting to update the adaptor to generate better OOD images more broadly. Furthermore, the authors provide both extensive empirical evaluations and theoretical analysis which further support the validity of the CATOD methodology.

STRENGTHS:
Strength:
- This paper provides a timely analysis of the difficult problem of adapting generative models to OOD concepts which is often overlooked in recent literature.
- CATOD showcases both strong empirical performance (CLIP score and CMMD score) on a reasonable set of 25 OOD concepts.
- Additionally, the paper provides strong theoretical links between the introduced aesthetic/concept-matching score and their necessity to the performance of adaptors.

WEAKNESSES:
The primary concern with this paper is the sparse empirical evaluations presented for CATOD. In particular, the 25 OOD concepts chosen seem few with only a small set of 100 samples left out for validation.

QUESTIONS:
The reviewer would like to have further clarification regarding the experimental setup of the paper. In particular, the reviewer would like to hear the author more directly address whether the small evaluation set used for the experiments is a reasonable concern.

LIMITATIONS:
The authors don't provide any full discussion of the limitations of the proposed methodology. Additionally, no direct comment has been made regarding societal impacts, however, the reviewer does not see any potential negative impacts derived from this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12