Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the-Wild
Authors: Jianan Yang
Chenchao Gao
Zhiqing Xiao
Junbo Zhao
Sai Wu
Gang Chen
Haobo Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Genearation
Text-to-Image
Diffusion
Out-of-Distribution
Active Learning
Primary area: generative_models

Abstract: The recent large-scale text-to-image generative models have attained unprecedented performance, while people established *adaptor* modules like LoRA and DreamBooth to extend this performance to even more unseen concept tokens. However, we empirically find that this workflow often fails to accurately depict the *out-of-distribution* concepts. This failure is highly related to the low quality of training data. To resolve this, we present a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD). Our framework follows the active learning paradigm which includes high-quality data accumulation and adaptor training, enabling a finer-grained enhancement of generative results. The *aesthetics* score and *concept-matching* score are two major factors that impact the quality of synthetic results. One key component of CATOD is the weighted scoring system that automatically balances between these two scores and we also offer comprehensive theoretical analysis for this point. Then, it determines how to select data and schedule the adaptor training based on this scoring system. The extensive results show that CATOD significantly outperforms the prior approaches with an 11.10 boost on the CLIP score and a 33.08% decrease on the CMMD metric.

Locating What You Need: Towards Adapting Diffusion Models to OOD Concepts In-the- Wild
Jianan Yang 1.2 Chenchao Gao Zhiqing Xiao Junbo Zhao
Sai Wul.2
Gang Chen Haobo Wang
College of Computer Science and Technology. Zhejiang University
Hangzhou Tegh-Tech Zone (Binjiang) Institute of Blockchain and Data Security
School of Software Technology. Zhejiang University
International School of Information Science and Engineering, Dalian University of Technology
{jianan0115, zhiqing xiao, zhao, wusai, cg, wanghaobo Heigghaoboeezjuu Jzzju. edu. gccgyllypggmail. com
Abstract
The recent large-scale text-to-image generative models have attained unprece- dented performance, while people established adaptor modules like LoRA and DreamBooth extend this performance even more unseen concept tokens. However, we empirically find that this workflow often fails to accurately depict the out-of-distribution concepts. This failure is highly related to the low quality of training data. To resolve this, we present a framework called Controllable Adaptor Towards Out-of-Distributic Concepts (CATOD). Our framework follows the active learning paradigm which includes high-quality data accumulation and adaptor training, enabling a finer-grained enhancement of generative results. The aesthetics score and concept-matching score are two major factors that impact the quality of synthetic results. One key component CATOD is the weighted scoring system that automatically balances between these two scores and also offer comprehensive theoretical analysis for this point. Then, it determines how to select data and schedule the adaptor training based this scoring system. The extensive results show that CATOD significantly outperforms the prior approaches with an 11.10 boost the CLIP score and 33.08% decrease the CMMD metric.
Introduction
The generative modeling for text-to-image has attained unprecedented performance most recently B7 45 49. Notably, by training over billions of text-image data pairs [52] [1], the family of diffusion models has allowed high-fidelity image synthesis directed by the prompt provided production. Despite their massive successes, these models still fail to generate images with decent quality and matching semantics, when encountering prompts that contain unseen or
concept tokens [60] 12 21 Simply put, the reason causing this failure is due to that the training set is not unbounded with limited variations. On the side of production, this limitation could significantly impact the practicality of this technique in real- wolld applications.
To deal with such concepts, recent works have resorted to adaptors such as Textual Inversion DroahBooth [7 58 48), and LoRA [25] 66 766,, which tunes only small part of the text-to-image model insert extra modules. These adaptors largely reduce the training costs, and more importantly, preserve the visual aesthetic information originally learned by the underlying model.
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Asodoti (rMam s2g
Filled Lawed (eMan a ity
Aquua (mosan neay
Alliganie Lizued (eNtaix huag
Whiscal (Chnd uss
Eugarer Pengain Chick 20US4 4334
Penguin (cand une.
RodiHlaperr
King Pesgutz Chock (cabaD) so00
Magellasic Pengutn (camat:
Adele Panguin
tesnin 6.m
Roal Image
Beloee
Adagting
Alkee
Adageing (LoRA)
Figure 1: Comparison of images generated before/after training adaptors over concepts with different CMMD scores. One observation is that concepts with higher CMMD scores are notably more challenging for the underlying model to generate (the second row). Additionally, we also notice that a higher CMMD value leads to more notable loss of visual details when training adaptors (the third row).
However, in this paper, we have found that recent works still struggle to accurately depict the visual details of dut-of-distribuutin concepts (with CMMD) score above 3.5), as illustrated in Figure] I Adapters like LoRA are able to accurately represent the shape and color of OOD concepts compared
to the generative results before adaptor training, but they fall short when it comes to finer details such as texture, contours, and patterns. This issue arises from the fact that current studies predominantly focus on variations of in-distribution (ID) concepts (e.g.. humans, dogs, and cats) while ignoring the out-of-distribution (OOD) ones. This failure motivates us to think about what makes the problem of distorted visual details happen when training adaptors.
In Figure 2 we observed that how an adaptor depicts OOD concepts can be significantly influenced by the quality of the training data. If the model is trained on samples containing disruptive objects, the resulting generative outputs are likely to reflect these disruptive elements. When the training data contains images with vague or very small instances of the OOD concept, the generative results may appear low-quality. In contrast, the high-quality data for adapting OOD concepts usually contains single and clear object corresponding to the given concept, which is highly distinguishable from the background and other types of objects and helps produce accurate results with high-fidelity. However, manually picking such high-quality data requires much human labor and expertise, which may crucially limit the versatility of text-to-image models. Therefore, an effective paradigm to locate high-quality samples of OOD concepts is important for the practical shipping of this field.
To this end, we developed a framework called Controllable Adaptor Towards Concepts (referred to as CATOD) which aims to identify high-quality samples to guide the adaptor training. This framework follows the Active Learning (AL) paradigm (54 involving iteratively accumulating training data and updating the adaptor. The profound motivation of this approach is to comprehensively model the interaction between training data and the underlying cext-to-image model. Specifically, CATOD includes two interconnected scores: the aesthetic score and the concept- matching score, following the observation that object clarity and uniqueness largely impact generative results, as illustrated in Figure 2 Based on this, we devised weighted scoring system that adapts itself according to the adaptor to select high-quality data while also properly balancing the two scores. With the carefully selected high-quality data, we schedule the adaptor training based on the quality of generative results evaluated through this scoring system.
In summary, our contributions are as follows: (i)-We have identified the challenge of adapting text-to-image models to out-of-distribution (OOD) concepts, where recent studies often struggle to accurately depict them; (ii)-We We have introduced a framework called CATOD that iteratively updates training data and the adaptor to generate OOD concepts precisely; (iii)-Our extensive experiments verified that CATOD achieves significant performance gain with up to 11.10 on the CLIP score and 33.08% on the CMMD metrie; (iv)-We We have also offered theoretical insights into the key factors: aesthetics and concept-matching which contribute to the effectiveness of our method.

---

Fed Sumodio Tostaal Lonosiiee
Lokks
Figure 2: Comparison of synthetic results on data with different quality. Generated images are significantly influenced by the quality of training data. If the training data includes disruptive objects, the generative images may include disruptive visual details (Left). When an object within the image is too small, the results may not accurately represent the intended concepts (Middle). In contrast, if the image contains a high- fidelity object without disruptive elements (Right), the model is more likely to generate the desired result accurately.
2 OOD Problem in Latent Diffusion Models
Revisiting Latent Diffusion Models (LDMs). Latent Diffusion Models (LDMs) 45 comprise two components: a diffusion process operating the latent space and an auto-encoder which contains an encoder & mapping an image into the latent space and a decoder D that reconstruct images from latent codes. Furthermore, the diffusion process can be conditioned on the output of text embedding models, enabling the auto-encoder to integrate the information derived from texts. Let x be the image, the CLIP textual encoder Co that maps the corresponding text y into the latent space, the LDM loss is:
LLDM (z, y) := [E==E(a),e-- [||e Eo (E,, t, co (v)) [12
(I)
where denotes the time step, 24 denotes the latent code noised at time step when e, cq represents the noised samples and the denoising U-Net [46], respectively. Through this procedure applied to the latent codes, LDM enables the underlying model to integrate information
derived from texts into the visual results, while also allowing more flexibility to produce images.
The OOD Concepts for LDMs. Intuitively, out-of-distribution (OOD) concepts refer to the category of data whose distribution deviates significantly from what the model has learned. This degree of drifting can be quantified by MMD score (27], which evaluates the discrepancy between ground tround-truth images and the generative results in the image latent space. Formally, for two probability distributions P and Q, the MMD distance with respect to positive definite kernel k is:
dist M M D (P,Q)= = Exy-Pas [k(x,, x²)] + Be R [M<(r.y 2//]] -2E,,-PP [K( pp F Rg)],
(2)
where Tpp xi, independently follow the distribution P while Ty o' independently follow the distribu- tion Q. with l: the Gaussian RBF kernel [17] In our implementation, we sample two sets of vectors from the distribution P and Q, then use CLIP embeddings (40] to calculate this score, which is also named as CMMD (27] We set concept with a high CMMD score (above 3.5) an OOD concept.
OOD Concepts are Hard to Adapt. Recent text-to-image LDMs 45 39 have achieved unprece- dented performance on a wide range of concept tokens. However, we have found that there still exist many concepts that make LDMs fail after the adapter fully trained. As we show in Figure T. there are several discoveries: (i)-The concepts with higher CMMD scores are much more challenging for the underlying model to generate or adapt. The concepts with a CMMD score above 3.0 show explicit wrong visual details. For Axolotl and Frilled Lizards with CMMD above 4.0, the LDMs even generate the wrong species; (ii)-A higher CMMD score indicates more severe loss of visual details when training adaptors. The concepts with CMMD scores above 3.5 in Axolotis and Emperor Penguin Chicks, show explicit distorted visual details, like color, texture, and delicate details. For Axolotl, the generative results show wrong number and wrong positions of amateurs. For Emperor Penguin Chicks, the generative results show the wrong fur color of their heads and wings. To summarize, the higher the CMMD score of a concept, the more difficult it is for LDMs to adapt.
The High-Quality Matters. We further observe that the generative results are quite sensitive to training data when training adaptors meet OOD concepts. As shown in Figure [ when the training images contain disruptive elements, the visual features of these disruptive elements will be easily introduced into the generative results. For example, when an adult emperor penguin appears in training data, then the black fur on their back can easily appear when generating their chicks, despite that their chicks have white fluffs shown in the left part of Figure 2 If the object of the desired
3

---

Adaptor Mloduie
Dillusion Model
Generetive Results
Saxted Lest
Aesthetle booce
Finestume
Scoctel Thiining Sangles
X
Wirghted Suveer
Data Space
St pengain
Cyce
X2
Query Secce
Doout
Concere Matchiin Store
Scoring Systemn
Figure 3: The overall pipeline of CATOD. brief, CATOD alternatively performs data selection and scheduled OOD concept adaption. In each training cycle, we first generate OOD concepts according to the current adaptor and calculate the weights for the aesthetic score and concent-matching score. Then, we calculate the weighted for each sample within the data pool pooo, select the top images accordingly, and add them the training pool. At last, CATOD fine-tunes the scoring system and training adaptors according to the updated data pool, and proceed to the next cycle. The above three steps alternatively proceed until convergence.
concept appears to be too small within the image, then the generative results tend to be bad-looking, since the necessary visual details are not fully identified. In the middle part of Figure we can see that distorted visual details like texture and shape in axolotl and emperor penguin chicks largely harm the aesthetics of the generated image. To generate images correctly and good-looking, we require enough amount of images with objects of high fidelity and do not contain disruptive elements, namely High-Qualin Quality samples as shown in the right part of Figure 2 Therefore, we aim to devise an effective data selection strategy for locating those high-quality images for these OOD concepts.
3 Method
3.1
Overall Architecture
We aim to adapt OOD concepts LDMs correctly by an iterative data selection criterion that locates high-quality data and training adaptors accordingly as shown in Figure 3 Consequently, our method would alleviate issues introduced by disruptive elements, e.g., irrelevant objects, and blurry images, while maintaining high fidelity of generative results. The core to CATOD is a scoring system, which consists of an aesthetic scorer and concept-matching scorer, aiming resolve the problems of incorrectly introduced visual details and distorted objects we have observed Section 2 By properly trading off between these scores, CATOD locates the most valuable samples for adapting underlying LDMs to OOD concepts and making the desired OOD concept depicted correctly.
3.2 The Scoring System
As described above, we need to estimate the potential impact of real-world samples over LDMs, according to which we select the most high-quality samples for training. In Section 2 we have observed two major factors that significantly impact generated image quality: object clarity and disruptive elements. Diving into the loss term LLDM(I,)) 3) Eq. (T) we may also observe that the training set Xς should be optimized towards both the underlying model (including the embedded adaptor) and the conditional text y, which indicates two important factors: object clarity and concept- matching achieved by preserving aesthetic information originally learned by LDMs and accurate image-text text matching, respectively. Therefore, we attribute the quality of images to aestheric and concept-matching and design decoupled scorers accordingly.
The Aesthetic Scorer. Aesthetic evaluation is long-standing field, which comprehensively considers whether the lighting, contrast, texture, and other photographic factors of an image consistent with

---

human aesthetics. The general aesthetic scorer can be simply described as p p- Sess where Saes indicates the aesthetic scoring model. p denotes the predicted score, and x represents the input image. Following the work of PA-IAA 33 we fine-tune the generic aesthetic model and make it adapt to OOD concepts with personalized preference scores. In general, we assign a high score to the samples within the training set Xr. assign a low score to samples from irrelevant categories and samples generated by underlying LDMs without an adaptor, and use them to fine-tune the aesthetic scorer. More details of this personalization are given in the Appendix B.3
The Concept-Matching Scorer. Intuitively, concept- describes whether the OOD concepts get perfectly reflected in the generated results. A similar task is image retrieval, which is designed to retrieve images containing objects describing the desired concepts. Since image retrieval also relies on feature maps and some sort of matching score to retrieve images, we adopt matching score from 5.5 as follows:
VLeD-rellted works
Xr1
Scon (x)
Σ
Φ (re, (e) exp (llr= rxll),
[
where rk), if r4k is the closet representation for T2 and is set to 0 otherwise. Simply put, VLAD regards the extracted representations for XT as a codebook and maps each image to its nearest code. Note that samples within Xy mostly consist of clean and clear samples, this score is sufficient to quantify whether the object if exists in the given image distinguishes itself from other photographic components and matches the given concept.
3.3 Active Data Acquisition
Optimization with Active Learning (AL). Aiming to mitigate the problems caused by low-quality training samples, we proactively integrate the training data X into our objective as follows:
A X; arg mix, Ez~x, LLDM (z, A. y).
Since the optimal set is initially unknown, one-step optimization can easily lead to convergence to local optima. Therefore, we iterative paradigm optimize adaptor A and training data
Xy. respectively. In our implementation, we adopt the paradigm of AL 44 (54] to perform the
optimization of x by data accumulation before training adaptors, with denotes the time step:
(B()__
arg min
E xx (ex LUBLL A,v),
BCD poo! -X 1,16=bb
where b is the number of samples added the training pool at each cycle. The main reason for using AL is its preferred sample efficiency, with better controllable data bias management [5 50) Instead of repeatedly selecting data from the whole real-world data pool, AL provides more efficient procedure to optimize training data by using data selection to accumulate high-quality training data. To this stage, the learning procedure of CATOD is relatively clear: the sample pool is progressively accumulated -by Eq. (5) -and the optimization of the adaptor A is straightforward (Fig. 3)
Remember that AL involves iteratively updating the adaptor and the training data, it is important to design a training schedule for adaptors and determine how to acquire high quality based on the two scorers mentioned above. The primary objective of this design is to achieve a dynamic trade-off between the two scores. The specific details of these designs are described below.
The Active Schedule for Training Adaptors. The training schedule has been found crucial for successful adaption [47 74 65). Since our training data continuously expands as the learning cycle of AL proceeds, the training schedule will be even more important. To arrange this schedule, we first calculate the aesthetics score ((ees(A) =
[xx(x Exr) (aes(T)) and concept-matching score Teon (A) 194 (X,)] Ex,Ens (X,) Scon (xg) of the adaptor A based on its generative results
gA (Xr), in which both the aesthetic score Tacs (A) and Yoon (A) range from to 10. Then, we use a trigonometric indicator that comprehensively measures its performance:
(4) = 10 sin
π 20 Tacs(()) (A)
sin
π
Trom Trom(()) (A))
20
Notably, (A) peaks when the two scores get close or around the common value of 5.0 for most samples. Mcanwhile, it bottoms when the scoring exhibits stronger signal of being biased (to either
5

---

side). The properties emphasize that adapters balancing the two factors are of better quality. In training CATOD, we use ((A) as a signal to reduce the learning rate and stop training in time.
Trading-off the Two Scores in Data Acquisition. After getting the adaptor A, we continue to select the most suitable samples for the next-cycle training. Notice that whether newly selected images enhance the adaptor depends on both aesthetics and concept-matching we ought to adjust our preferences according to the adaptor. In more detail, when the adaptor has high aesthetics but does not accurately depict the OOD concept, samples with high concept-matching scores better enhance the adaptor; when the adaptor fails to exhibit photographic attributes consistent with humans, samples with high aesthetics are preferred. To implement this preference, our acquisition score is:
π ((x) = (1 sin
π 20 Jaes (4acs(4))) Saes (2) + (1 sin 22 Yeon (A)) Soon (1)).
This formulation offers some meritable advantages. On one hand, the balancing terms are not pre- fixed manually tuned, but dynamically dependent the scores marginalized over the current generations. Further, when the score of either side gets larger, the corresponding balancing coefficient, in turn, decreases, thus attaining proper trade-off. As result, this mechanism encourages an alternation of sample selection towards both scoring metrics by selecting Top- samples to add to the training data XT accordingly, which effectively guarantees the sample diversity.
4 Theoretical Insights
This section presents our theoretical analyses of why aesthetic/ooncep--aatciii scores work in OOD Adaption. Specifically, we derive the distance between real- world data distributions and synthetic data distributions and then induce the important factors that affect this distance. We first introduce the minimum mean square error (MMSE) [22
to measure the discrepancy between distributions:
Definition 4.1. The minimum mean square error (MMSE) of estimating an input random vector X E R. from an observation/output X E RRt defined as
MMSE inf JEM(()))) EE
||- [(X)
in which (RR) denotes the space consisting of all measurable functions on R".
Notice that we are trying to produce the best generation results which are initially unknown, our adaption task can be also regarded as estimating the optimal distribution with carefully selected data. By denoting the ideal generative results and the ideal adaptor with random variables XG and A", respectively, we conclude that the LDM loss LLDM is consistent with this MMSE term:
Theorem 4.2. Let LLDM be the LDM loss following Eq. D and the image space lies within IR.. Then there always exists an ideal random vector XG E IR" and an adaptor A for LDM, such that
arg min (MSX(Xx(XT)) Xy
arg min Exwx, [LLDM Az)
XyeRR
Based on the preceding deduction, we have confirmed that the change MMSE can also indicate the change in LDMs. To dive deep into the MMSE term, further decompose it as follows:
Theorem 4.3. (Pythagorean Theorem for MMSE (I3].) Following Theorem 4.2 by setting f to the generative model JA, the MMSE term in Eq. (8) can be decomposed into two terms as follows:
E & (x(
E
& ga- (x)) [1ga- (XT) (X+))l
(10)
in which A" denotes a potential ideal adaptor containing information for text y. Notice that the generative results X relies on the training set X7, Ia (X7) can also be written as Xx
Upon revisiting the two terms, we can gain deeper understanding of the relationship between MMSE and besthetic/concep--matchi score: (i)-The first term is focused on estimating the difference between the generative distribution X XT and the ideal distribution of Xg. This assessment helps

---

Table 1: A Comparison over the performance of CATOD, in terms of the CLIP score and CMMD score with 100 images sampled last. This table shows the average result of sub-classes within each category. The overall improvement of our proposed CATOD is provided by "Imp.". Methods with the best performance are bold-folded.
Comparison Methods
insect
lizard
penguin
CLIP Scurc (1)
seafish
snake
Avg.
Imp.
insect
lizard
penguin
CMMD 22] (1)
seazish
stake
Avg.
Imp.
DrothBooth
57
RAND
65.35
67. 39
68.07
65.59
72.07
67.79
17.58
1.35
1.39
1.67
1.59
1.25
1.45
(00550
DreumBooth
57
CLIP
20.56
72.19
72.09
71.17
74.59
72.12
13.25
1.04
1.24
1.05
1.38
1.16
1.17
(0.22
DreumBooth
CATOD
72.18
75.30
75.16
74,60
79.61
75.37
0.80
0.74
RAND
61.35
17.04
2.45
2,89
CLIP
61.65
67 24
66.95
62:27
64.23
64.47
13.51
2.11
1.86
1,54
1.87
1.37
1.73
(00.30
Ti TM
CATOD
69.21
70.14
68.23
64.89
67.41
67.98
1,57
1.73
1.15
1.43
1.25
1.43
LoRA 73
RAND
64.39
65 02
67.49
68.8
71.09
67.57
110.60
1.52
1.47
1.56
1.61
1.18
1.47
(0.3
LoRA
CLIP
20.27
74,13
72.08
73.19
75.64
73.06
14.91
1.29
1.33
1.04
1.35
0.89
1.18
00.34
CATOD
72.60
77.00
74.11
84.29
0.71
0.77
us understand how the introduced samples lose visual information within the underlying LDM, thus we can connect this term to aesthetic preservation, i.c. acsthetic score; (ii)-The second term illustrates the degree to which the training set XT distorts the OOD concept information within the ideal adaptor. Hence, a concept-matching score accurately portrays how newly given samples impact this term. At this stage, we have completed the theoretical support showing that both aesthetics and concept- concept-matching major factors that influence the performance of adaptors.
Experiments
In this section, we present the main experimental results both qualitatively and quantitatively. To evaluate our proposed CATOD, we combine it with several works for adaption, i.e. DreamBooth [47). Textual Inversion 18), and LoRA (25). More experimental results can be found in the Appendix. The source code is attached in the Supplementary:
Datasets. We test our method on datasets with 25 OOD concepts that can hardly be generated through prompt engineering on the text-to-image model. This dataset consists of 5 categories: insect, lizard, penguin, seafish, and snake, and each category contains data from 5 OOD concepts. Each concept has 1,000 examples in total with 100 samples left out for validation. The dataset is collected from publicly available datasets including ImageNet, iNaturalist 2018 667, IP102 [7
Implementation Details. We conduct the active generation experiments on our proposed CATOD and three representative adaptors, i.e. DreamBooth 47 Textual Inversion [18] (termed TI in the paper). and LoRA (25). Since there are currently no available studies that focus on locating "high-quality" samples for training, sampling (CLIP) for each baseline in our active learning setting. Each experiment starts with 20 randomly sampled instances, and we conducted 5 cycles of data accumulation in which we selected 20 "good" samples to add to the training pool. We train 20 epochs for all combinations of adaption techniques and sampling strategies in each active learning cycle, with batch size of 1. Furthermore, we generate 100 images for each concept for evaluation. We use the commonly adopted Stable Diffusion 2.0 pre-trained on LAION-5B (51] following Rombach's 's work 455
apply random sampling (RAND), and CLIP-score-ban -ased
Evaluation Metrics. We evaluate the quality of our generated images with the widely used CLIP score (20 and the recently proposed CMMD score (27), which quantify the model performance in two aspects. Specifically, the CLIP score measures how generated images match the given text, which is expected to be high as possible. Meanwhile, the CMMD score evaluates the discrepancy between generated images with the real ones, indicating better generative results with lower values.
5.1 Single-concept Generation Results
To evaluate the performance of CATOD on OOD concepts, we test it on all 25 target concepts one by one separately and report the average performance of 5 concepts within each category in Table [ We show the superior results of our CATOD with respect to both qualitative and quantitative comparisons.
Qualitative Comparisons. We qualitatively compare CATOD with other sampling strategies accord- ing to their generated images based on LoRA (55). With random sampling (RAND), we observe that the generative results only partially learned some photographic attributes like color and texture,
7

---

Table 2: A Comparison the performance of CATOD when training with images from multiple concepts, in terms of the CLIP score and CMMD score with 100 images sampled at last. In each experiment, we sample images from all the sub-classes within each category and check whether the fine-tuned model can generate all 5 concepts. The overall improvement of our proposed CATOD is provided by "Imp.". Methods with the best performance are bold-folded.
Comparisen Methods
inseet
lizard
penguin
CLIP Sioore (1)
scafish
snake
Asrg
Imp.
insect
lizard
penguin
CMMD 27 (8)
scafsh
snake
AAgg.
Imp.
DreamBooth
++
RAND
63,29
63.79
65.72
65.59
64.36
64.55
(18.0
1.74
2.14
2.16
2,02
1.85
1.98
L0.75
DreamBooth
CLIP
20.35
70.38
11.00
1.53
1.80
(0:46
DreamBooth
27
CATOD
70.83
72.28
74.31
70.90
75.45
72.75
1.39
1.25
1.34
1.45
0.73
1,23
T
RAND
59.23
56.97
$7.90
60.83
62.65
59.52
(1.88
2.84
2.56
2.27
2.39
2.41
2.49
10.83
CLIP
61.74
60.72
63,79
62:71
65.71
02:95
(2.47
2:26
2.08
1.94
1.97
2.25
2.10
10:44
11
CATOD
64,48
63.93
65.93
65.44
07.24
65.40
1.76
1.53
1.65
1.64
1.73
1.66
RAND
66.65
67.64
1.79
2.05
1,93
U0:59
LoRA
CLIP
69.25
70.84
11.39
71.91
72.78
71.23
13.06
1.40
1.09
1.74
1.59
1.28
1.54
(0.30
LoRA
CATTD
71.19
74.09
73.68
75.60
76.90
74.29
1,13
1.37
1.49
1.26
0.95
1.24
Cround Truth
RAND
CATOD CLIP
Figure 4: A comparison of different sampling strategies with LoRA. Specifically, compare three lines of works: (1) RAND, in which the model is trained with 100 randomly selected samples: (2) with samples of the highest CLIP scores (100 samples); (3) 100 samples with CATOD. The model trained with randomly sampled data fails to capture the features of of-distribution (OOD) concepts, while the ones trained with top CLIP scores contain necessary details but also include disruptive elements.
but failed to make the objects have the correct appearance and shape. CLIP-based sampling (CLIP) somehow produces the corrected shape of the concept but still fails to capture the necessary details for describing the object. comparison, our proposed CATOD successfully matches all the photographic attributes, while also guaranteeing the image aesthetics.
To further look at how CATOD learns the photographic attributes that precisely match the concept, we show samples generated from different cycles in Fig. 7 We can see that some attributes like color, and texture are already learned in cycle I, but the shape of the object does not match the ground-truth ones. From cycle 1 to 3, CATOD shows a clear shape modification, making the objects more like the real ones. From cycles to 5, an iterative refinement on more photographic details like light, contrast, and other minor modifications (like the beak for penguins and antenna for axolotis) is shown in generative results, making them hard distinguish from the ground-trum -truth ones even with a careful look. At cycle 5 and later cycles, the image quality stabilizes and we can hardly see enhancement apart from image diversity. To conclude, we can see an explicit attribute matching process from easy ones to the finer ones within CATOD, showing the importance and effectiveness of iterative training.
Quantitative Comparisons. Table [ reports CLIP scores (20] and CMMD scores (2 from each strategy with models trained on 25 different OOD concepts and evaluated through the generative results. Specifically, we evaluate the performance of CATOD on each concept and average the results within each category in T We have the following conclusions: (1) the average performance of our strategies outperforms all compared methods by 0.56~11.10 CLIP score, justifying the effectiveness of locating aesthetic and clean samples over text-image matching. (2) CATOD also brings a 0. 12-0.554 CMMD decrease on various frameworks and concepts, indicating better image alignment with proper sampling guided by CATOD. To summarize, both image-machhing and text- matching scores exhibit better results compared to the baselines, suggesting that our proposed CATOD significantly outperforms other strategies, which is consistent with the qualitative results in Fig. 4
8

---

Grond Truth
(Unseen Concept) Cycle 1
Cycle
Cycle a
Cycle
Cycle
Cycle 6
Cycle
Figure 5: Generative results as cycle proceeds. Samples are generated CATOD on cycles from To better observe how generated images change as the cycle proceeds, we conduct another 2 cycles here. In each cycle, we select and add 20 high-quality samples. Generative samples start to converge and contain the right details within the original concept after cycle 4 We also that those generative results contain diverse contents within the background based on the few images given.
Table 3: Results Ablating Aesthetic, Concept- Matching Scorer and Weighted Scoring on CATOD. We show the average results conducted on the categories "pen- guin" and "lizard" with LoRA.
Table 4: Results of CATOD with different types of aesthetic scorers. We show the average results conducted on the categories "penguin" and "lizard" with LoRA.
Modules
CLIP(T)
CMMD())
Concept
Weighted Scoring
Type Aestbetic Score
CLIP(")
CMMD()
Matching
penguin
penguin
lizard
penguin
lizard
penguin
73.19
72.85
1.14
1.28
ReLIC ST
71.35
72.39
1.29
1.59
1.10
TANet 23
72.05
72.67
1.32
1.35
75.35
73.24
0.94
0.93
BAID 72
73.08
72.79
1.18
1.37
77.00
74.11
0.89
0.71
Ours
73.19
72.85
1.14
1,28
5.2 Multi-concept adaption Results.
To further investigate whether CATOD could adapt multiple concepts simultaneously. we group the 25 concepts by category and train the adaptation model on each category. To be specific, we compare our baselines using a generated set consisting of 500 images (100 images per concept) and exhibit our results in Table 2 Following the setting of single-concep adaption, we conduct 10 cycles of data accumulation and get 200 samples last, since colci-c adaption requires more data. We still observe a notable performance gain with a 1.56~5.007 CLIP score increase and O. 12~0..86 CMMD score decrease compared baselines. These results verify that our CATOD still achieves better performance on multi-concept adaption.
5.3 Ablation Studies
We verify the efficacy of all components in our proposed CATOD in Table and including the aesthetical scoring module and the concept-matching mechanism within the weighted scoring system.
W/O Aesthetic Score. First, validate the effectiveness of CATOD by removing the aesthetic scores. significant decrease in performance (up to 8.55) on the CLIP score can be observed in Table We attribute this decline the fact that matching- based metrics prioritize image representations over the given concept, leading to a deterioration in image-text matching.
The Type of Aesthetic Scores. To further investigate the impact of aesthetic scores, we have also applied different types of aesthetics with CATOD in Table We observed that recent ofatc-of-the- art aesthetic evaluations did not improve OOD adaption and even led to minor performance loss. This could be attributed to the fact that these models were originally designed for general aesthetic assessments, whereas our aesthetic scorer is highly personalized towards specific OOD concepts.
W/O Concept-Matching Score. After removing the concept-matching scorer, we observed that the adaptor tends to perform better compared to just removing the aesthetic scorers. This might be due to aesthetic scorer being designed based on CLIP backbones, showing some consistency with the CLIP score. However, it still demonstrates limitations based on the image-matching score CMMD. While these aesthetic qualities partly describe the clarity and accuracy of the object, they not adequately focus the image representation space.

---

The Weighted Score. We have confirmed the effectiveness of balancing two scores by simply adding them together to guide CATOD (Table Line 5). We observed that this resulted in consistent performance loss for both the CLIP score (up to 1.65) and the CMMD score (up to 0.22). This suggests that both text-image matching and image-to-image matching are affected by the trade-off between aesthetics and concept matching, further emphasizing the importance of these two scores.
6 Related work
The task of text-to-image generation 79 and has achieved impressive Therefore, adapting large-scale text-to-image models to specific concept while also preserving this amazing performance, i.e. personalization [7], has become another recent research interest. But this is often difficult since re-training model with an expanded dataset for each new concept is prohibitively expensive while fine-tuning the whole model [12 32 or transformation modules [83] 20
Personalized Text-to-image Synthesis with Adaptors.
involves creating specific images based on text descriptions 3
73
performance with state-of-the-arr diffusion models 41
45
39
60 on few examples
typically leads some rate of forgetting 30 Therefore, adaptors such Textual Inversion [18),
DreamBooth (4 48
5 [88, LoRA
25
66 86 (76). along with some other works 70
19 derived
from them, have become more commonly adopted. Typically, they focus on small but erucial part of the model or extra networks inserted into underlying models, thus more computationally--ffiiies while also preserving the efficacy of the underlying models with lower computational costs. For example, textual inversion (TI) [88 represents the newly-given concept with pseudo word [42] and remapping another carefully trained embedding in the text-encoding space, guided by few images. Despite their computational efficiency, these approaches are still facing difficulties dealing with -fitt--distribution concepts as we observe in Section 2
Active Learning and Selection. Active Learning is a machine learning paradigm that involves actively selecting the most suitable data for training models from external data sources [4] 63]. The most crucial part of active learning is the strategy to locate the optimal data batch. Current studies can be roughly categorized as follows: (a) Score-based methods that prefer the samples with the highest methods searching for the samples that the most representative of the underlying data distribution [53] T 62]. The Active Selection paradigm within Active Learning serves as an efficient and powerful dataset curation tool (i.e. adaption), leading
information scores (36] 69 [0]: (b) Representation-basse
to numerous studies adopting this paradigm from a wide range of subjects [68 24, g. Meanwhile, due the effectiveness of Active Learning in selecting the most suitable training samples, recent studies have utilized similar sampling strategies to address challenges associated with long-tailed distributions (56] [57] and noisy data (72). Given that Oftttr-Distiibttioon (OOD) concepts in Latent Diffusion Models (LDMs) often involve unseen or long-tailed concept tokens [79], this motivates us to leverage Active Learning for selecting samples that are well-suited for training adaptors.
In this work, we focus on scored-based strategies in two-fold: aesthetics and concept-matching Image Aesthetics Assessment (IAA) aims evaluating image aesthetics computationally and automat- ically (9) while automatically assessing image aesthetics is useful for many applications [35 29 14).
To take step further, personalized image aesthetics assessment (PIAA) 43 85 75)
was proposed to capture unique aesthetic preferences, consistent with our goal to adjust our paradigms accordingly different concepts. At the same time, "Concept-matching" in which we search for relevant images in an image gallery by analyzing the visual content (e.g.,
adopted from the field of image retrieval,
objects, colors, textures, shapes etc.), given a query image [6]] BT To design a paradigm that automatically meets the harsh requirements for adaptor training, we consider both two factors.
7 Conclusion
We propose CATOD, an enhanced, data-efficient, and practically useful version of the OOD concept adaptation for AIGC. This method is encapsulated in an active-learned paradigm with carefully designed acquisitional scoring mechanisms. CATOD significantly outperforms the prior approaches in many (if not most) aspects including generation quality, concept matches, technological robustness, data efficiency, etc. In the future, we hope to ship CATOD to the open-source community so as to absorb more OOD concepts that were originally uncovered.
10

---

Acknowledgements
This work supported by the Pioneer R&D Program Zhejiang (No. 2024C01035), This work is also partially supported by the Fundamental Research Funds for the Central Universities (226-2024- 00049) and (26-5004-000145).
References
[1] Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[2] Song Bai, Xiang Bai, Qi Tian, and Longin Jan Latecki. Regularized diffusion process on bidirectional context for object retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, (1(5):1213-12) 2018.
[3] Razan Bayoumi, Marco Alfonse, and Abdel-Badeeh M Salem. Text-to-image synthesis: A comparative study. In Digital Transformation Technology: Proceedings of ITAF 2020, pages Springer, 2021.
[4] Pu Cao, Feng Zhou, Qing Song, and Lu Yang. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04277 044449900049 2024.
[5] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Dis- enbooth: Disentangled parameter-effiiient tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03374 2023.
[6] Edwin K. P. Chong. Well-conditioned linear minimum mean square error estimation. IEEE Control. Syst. Lett., 6:2431-2436, 2022.
[7] Niv Cohen, Rinon Gal, Eli A Mcirom, Gal Chechik, and Yuval Atzmon. "this is my unicorn, fluffy": Personalizing frozen vision-language representations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX, pages 558 577. Springer, 2022.
[8] Wupeng Deng, Quan Liu, Feifan Zhao, Due Truong Pham, Jiwei Hu, Yongjing Wang, and Zude Zhou. Learning by doing: A dual-loop implementation architecture of deep active learning and human- machine collaboration for smart robot vision. Robotics and Computer-Integrated Manufacturing, 86: 102673, 2024.
human-machine
[9] Yubin Deng, Chen Change Loy, and Xiaoou Tang. Image aesthetic assessment: experimental survey, IEEE Signal Processing Magazine, 34(4):80-106, 2017.
[10] Francesco Di Fiore, Michela Nardelli, and Laura Mainini. Active learning and bayesian optimization: a unified perspective to learn with a goal. Archives of Computational Methods in Engineering, pages 1-29, 2024.
[11] Mario Díaz, Peter Kairouz, and Lalitha Sankar, Lower bounds for the minimum mean-squad error via neural network-band based estimation. CoRR, abs/2108. 12851, 2021.
[12] Yuxuan Ding, Lingqiao Liu, Chunna Tian, Jingyuan Yang, and Haoxuan Ding. Don't stop learning: Towards continual learning for the clip model. arXiv preprint arXiv:2207.099 2022.
[13] Alex Dytso, H Vincent Poor, Ronit Bustin, and Shlomo Shamai. On the structure the least favorable prior distributions. In 2018 IEEE International Symposium Information Theory (ISTT), pages 1081-1085. IEFF, 2018.
[14] Yuming Fang, Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3677-3686, 2020.

---

[15] Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. arXiv preprint arXiv:2101.11663 2021.
[16] Li Fe-Fei et al. A bayesian approach to unsupervised one-shot learning of object categories. In proceedings ninth IEEE international conference on computer vision, pages 1134-1141. IEEE, 2003.
[17] Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf. and Bharath K Sriperumbudur. Charac- teristic kernels on groups and semigroups. Advances in neural information processing systems, 21, 2008.
[18] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv:2208.016/3 2022.
[19] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 113, 2023.
[20] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma. Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:2 10.04544 2021.
[21] Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv:2307. /2980, 2023.
[22] Dongning Guo, Shlomo Shamai, and Sergio Verdú. Estimation of non- gaussian random variables in gaussian noise: Properties of the MMSE. In Frank R. Kschischang and En-Hui Yang, editors, 2008 IEEE International Symposium on Information Theory, ISIT 2008, Toronto, ON, Canada, July 6- 11, 2008, pages 1083-1087. IEEE, 2008
[23] Shuai He, Yongchang Zhang, Rui Xie, Dongxiang Jiang, and Anlong Ming. Rethinking image aesthetics assessment: Models, datasets and benchmarks. In Proceeding of the Thirty-First International Joint Conference on Artificial Intelligence, 2022.
[24] Aral Hekimoglu, Michael Schmidt, and Alvaro Marcos-Ramiro. Monocular 3d object detection with lidar guided semi supervised active learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2346-2355 2024.
[25] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models, In International Conference on Learning Representations, 2021.
[26] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2023.
[27] Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards a better evaluation metric for image generation. arXiv preprint arXiv:2401.0966 2023.
[28] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez. Aggregating local descriptors into a compact image representation. In The Twenty-Third IEEE Conference on Computer Vision and Patern Recognition, CVPR 2010, San Francisco, CA, USA, 13-18 June 2010, pages 3304-3311. IEEE Computer Society, 2010.
[29] Shu Kong, Xiaohui Shen, Zhe Lin, Radomir Mech, and Charless Fowlkes. Photo aesthetics ranking network with attributes and content adaptation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 662-679. Springer, 2016.
12

---

[30] Ananya Kumar, Aditi Raghumathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine- tuning can distort pretrained features and underperform nut-of-distriuttion. arXiv preprint arXiv:2202. 10054, 2022.
[31] Michael S Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM). 2(1): 1-19, 2006.
[32] Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu. Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. Proceedings of the 2022 Conference of the North Americun Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 5441-5454, 2022.
[33] Leida Li, Hancheng Zhu, Sicheng Zhao, Guiguang Ding, and Weisi Lin. Personality-assisted multi-task learning for generic and personalized image aesthetics assessment. IEEE Transactions on Image Processing, 19:3898-3910, 2020.
[34] Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. Advances in Neural Information Processing Systems, 32, 2019.
[35] Yiwen Luo and Xiaoou Tang. Photo and video quality evaluation: Focusing on the subject. In Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part III 10. pages 386-399, Springer, 2008.
[36] Xiaoyi Mai, Salman Avestimehr, Antonio Ortega, and Mahdi Soltanolkotabi. On the effective- ness of active learning by uncertainty sampling in classification of high- dimenssinnii gaussian mixture data. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4238 4242. IEEE, 2022.
[37] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 2rX1v:2112.10741, 2021.
[38] Michael A Nielsen. Neural networks and deep learning, volume 25. Determination press San Francisco, CA, USA, 2015.
[39] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 2rX.v:2307.01952 2023.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages $748-8763. PMLR, 2021.
[41] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.066 2022.
[42] Natalie Rathvon. Early Reading Assessment. A Practitioner's Handbook. ERIC, 2004.
[43] Jian Ren, Xiaohui Shen, Zhe Lin, Radomir Mech, and David Foran. Personalized image aesthetics. In Proceedings of the IEEE international conference on computer vision, pages 638 647, 2017.
[44] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CV Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
13

---

[46] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer- asssited intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 241. Springer, 2015.
[47] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[48] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wad- hwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models. arXiv preprint www.ii23377.06499 2023.
[49] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorcalistic text-to-image diffusion models with deep language understanding, arXiv preprint 1xXiv:2255.11487 2022.
[50] Hitesh Sapkota and Qi Yu. Balancing bias and variance for active weakly supervised learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1536-1546, 2022.
[51] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 08.0002, 2022.
[52] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fitered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
[53] Silvio Savarese Sener et al. Active learning for convolutional neural networks: A core-set approach. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.ner, 2018.
[54] Burr Settles. Active learning literature survey. 2009.
[55] Shihao Shao, Kaifeng Chen, Arjun Karpur, Qinghua Cui, André Araújo, and Bingyi Cao. Global features are all you need for image retrieval and reranking. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 11002- 11012. IEEE, 2023.
[56] Jiang-Xi Xin Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How ra-slinling helps for long-tail learning? Advances in Neural Information Processing Systems, 36:75669-75687, 2023.
[57] Jiang-Xin Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin- Yan Han, and Yu-Feng Li. Long- tail learning with foundation model: Heavy fine-tuning hurts. In Proceedings of the 41st International Conference on Machine Learning, pages 45014-45039, 2024.
[58] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instanthooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411 2023.
[59] Josef Sivie and Andrew Zisserman. Video google: A text retrieval approach to object matching in videos. In Computer Vision, IEEE International Conference on, volume 3, pages 1470-1470. IEEE Computer Society, 2003.
[60] Gabriel Skantze and Bram Willemsen. Collie: Continual learning of language grounding from language-image embeddings. Journal of Artificial Intelligence Research, 74:1201-1223, 2022.
[61] Aroold WM Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. Content-based image retrieval at the end of the early years. IEEE Transactions on pattern 2000.
analysis and machine intelligence,
14

---

[62] Qun Sui and Sujit K Ghosh. Similarity-basee active learning methods. Expert Systems with Applications. 151:123849, 2024.
[63] Alaa Tharwat and Wolfram Schenck. A survey on active learning: ofate-of-the-art, practical challenges and research directions. Mathematics, 11(4))820, 2023.
[64] Weijie Tu, Weijian Deng, Tom Gedeon, and Liang Zheng. A ofotoo--ptttpesss representation for dataset-level applications. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 2881-2892. IEEE, 2023.
[65] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-pin diffusion features for text-driven image-to-iman translation. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 1921-1930. IEEE, 2023.
[66] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computa- lional Linguistics, pages 3266-3279, 2023.
[67] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 8769-8778, 2018.
[68] Jinsheng Wang, Guoji Xu, Peng Yuan, Yongle Li, and Ahsan Kareem. An efficient and versatile kriging-basee active learning method for structural reliability analysis. Reliability Engineering & System Safety, 241:109670, 2024.
[69] Tianyang Wang, Xingjian Li, Pengkun Yang, Guosheng Hu, Xiangrui Zeng, Siyu Huang, Zheng Xu, and Min Xu. Boosting active learning via improving test performance. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8566-8574, 2022.
[70] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943- 15953, 2023.
[71] Xiaoping Wu, Chi Zhan, Yu-Kun Lai, Ming-Ming Cheng, and Jufeng Yang. Ip102: A large-scale henchmark dataset for insect pest recognition. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8787-8796, 2019.
[72] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang. and Yu-Feng Li. Nge: A unified framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 62-71, 2021.
[73] Weihao Xia, Yulun Zhang, Yujuu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. Gan inversion: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[74] Zeyue Xue, Guanglu Song, Qiushan Guo, Boxiao Liu, Zhuofan Zong, Yu Liu, and Ping Luo. RAPHAEL: text-to-image generation via large mixture of diffusion paths. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[75] Yuzhe Yang. Liwu Xu, Leida Li, Nan Qie, Yaqian Li, Peng Zhang, and Yandong Guo. Per- sonalized image aesthetics assessment with rich attributes. Proceedings of the IEEE/CV Conference Computer Vision and Pattern Recognition, pages 19861-19888 2022.
15

---

[76] Shin- Ying Yeh, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From lycoris fine-tuning to model evaluation. arXiv preprini arXiv:2309. /4859, 2023.
[77] Ran Yi, Haoyuan Tian, Zhihao Gu, Yu-Kun Lai, and Paul L Rosin. Towards artistic image aesthetics assessment: a large-scale scale dataset and new method. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2023.
[78] Mariia Zameshina, Olivier Teytaud, and Laurent Najman. Diverse diffusion: Enhancing image diversity in text-to-image generation. arXiv preprint arXiv:2310. 12583, 2023.
[79] Chenshuang Zhang. Chaoning Zhang, Mengehun Zhang, and In So Kweon. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.0790/ 2023.
[80] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[81] Lin Zhao, Meimei Shang, Fei Gao, Rongsheng Li, Fei Huang, and Jun Yu. Representation learn- ing of image composition for aesthetic prediction. Computer Vision and Image Understanding, 199:103024 2020.
[82] Yan Zhao, Lei Wang, Luping Zhou, Yinghuan Shi, and Yang Gao. Modelling diffusion process by deep neural networks for image retrieval. In BMVC, page 161, 2018.
[83] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348 2022.
[84] Hancheng Zhu, Leida Li, Jinjian Wu, Sicheng Zhao, Guiguang Ding, and Guangming Shi. Personalized image aesthetics assessment via meta-learning with bilevel gradient optimization. IEEE Transactions on Cybernetics, 1798-18811181111 2020.
[85] Hancheng Zhu, Yong Zhou, Leida Li, Yaqian Li, and Yandong Guo. Learning personalized image aesthetics from subjective and objective attributes. IEEE Transactions on Multimedia, 2021.
[86] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411, 2023.
16

---

A Theory
This section provides complete derivation for the analysis given in Section In brief, we first link LLDAS to minimum mean square ervor (MMSE) term in Theorem A.2 (i.e. Theorem 3.2 within the main context), then decompose the MMSE term see how we minimize MMSE different aspects in Theorem A.3 (i.e. Theorem 3.3 within the main context) in the main context. For convenience, we reclaim some of the formulations at the beginning of this section:
Definition A.I. The minimum mean square error (MMSE) of estimating an input random vector
X E R from an observation/output X RR. defined as
|| f(x)
|
17
MMSE ( (E EE
(11)
in which M(R)) denotes the space consisting of all measurable functions on R
Based on Eq. [ we can adapt the LDM an arbitrary concept with corresponding image set X that describes this concept. Typical adaptations are mostly based on fine-tuning all the parameters 8, which is quite costly and requires lot of data. However, recent studies have found that training only a small part of the model [18] [47] or inserting extra networks [25] 26] could attain the same performance, which largely alleviates the computational burden with far fewer samples need for training. We call this part of parameters or networks as "adaptors" and denote them by A. Then, the fine-tuning process of the adaptors, also named adaption, be formulated as follows:
A arg min (xx) [[le EB,A (Zt, l, Ca,A (v)) [
(12)
Therefore, be conditioning vector corresponding to some given text y, the LDM loss is:
LLDDM A, v) =
[lle 60.4 (Zz, L CO.A (V)) [13]
(13)
Following the definition of LDM loss, continue to link LDM loss the MMSE term in the theorem as follows (in which we omit the parameter y since it is independent of other parameters):
Theorem A.2. Let LLDM be the LDM loss following Eq. (1), and the image space lies within IR.". Then there always exists an ideal random vector &g E Ik and a condition vector 8² within the text embedding space for LDM, such that
arg min AMSE(X./XX) = arg min Enwxy (LLDM (x,, A*)] Er
XyER"
Proof. By denoting an ideal generative distribution with X which is produced by the generative model with a minimized LDM loss, we continue our proof. Set fa (Xr) Ex [z, A]. we can see that fA is also a functional controlled by the adaptor A. Following the definition of MMSE and a fixed Xr, we A as:
A arg min B,~X, [LLDM (2, A)]
(14)
With this A", and the Universal Approximation Theorem 38 for all deep learning models, fa- becomes the functional needed to quantify the MMSE term following its definition. Therefore, with proper XG, A", we have
xM Ex-xx [LDDS (s, A*)],
thus completing the proof of this theorem.
Theorem A.3. (Pythagorean Theorem for MMSE (3V.) Following Theorem A.2 by setting f to the generative model 9*a with an ideal adaptor A containing sufficient OOD concept information, rewrite gA (X)) to X XT, and the minimum mean square error (MMSE) in Eq. [T] can be decomposed into two terms:
E Ra
E E|x X
||
-E [| gn~ (X+)|l +E Ellgg-- (Xr) E [r]]],, (15)
17

---

Proof. This equation can be further decomposed into two terms:
E X - E
X Xrl 6(X))
= E [x (x))(Xx)) + I2 X - E X X Φ (X+))
Since X X represents the variance for generative results and is orthogonal to Xy, the first term is O, making it sufficient to consider only the second term. To prove that the second term is also 0, we first prove the orthogonality property, i,e.
E
(
&G
E E| o(x))
(16)
for any function 6.
E LE
X G 6(XT)
E X XT (2)) =I))
P(X Figs XT 0b)
((x))((xx= =x) 21)
p = 1z1)
(g P(XG = Fig =x=z,
25
E
[Xoo(X)) (X))
Therefore, we have the orthogonal property in Eq. (6).
Now we continue our proof for the main theorem. First, we can decompose Eq. 115 as follows:
& - E X X+]](
E
E
|8 24- (XT) + 9A- (XT) - B X [1a]
E [& 9A- 94- (XT) + [EAAA - E XT] III
2
X
94 (X+) (0. (XT) - E X X
Since the gap between pure generative result guided by the training set XT and an ideal text embedding is purely influenced by the training set X7, 9A- (Xr) E [Xa[Xr] can also be regarded as a functional over X2, making the last term equal to according to the orthogonal property in Eq. [6.. At this step, we obtain the result in Eq. 15
B More Details About CATOD.
In this section, we provide more details about our proposed Controllable Adaptor Towards Out-of- Distribution Concepts (CATOD).
B.1 An Overall Paradigm
Initially, we have a large data pool Dpoot related to the concept given with different quality and begin with an initial randomly-sampled training dataset x(() Xx might contain distorted images or even mismatch the given concept, which is common in publicly available datasets. Following the typical paradigm Active Learning, we train all the models we use on the training set Xy. Then, we select batch of data B from the data pool D pool according to the acquisition functions induced from models. Finally, we move this selected batch of data B to XT, go back to the model training step to re-train or adapt the models, and repeat this cycle until X reaches its maximum capacity or the quality of models cannot be further enhanced by adding new data. An overall Algorithm for cycle of CATOD is provided in Alg. [
18

---

Algorithm An active selection cycle for CATOD.
Text-to-image model g (-)-) that can be integrated with adaptor A, real-world data pool D poots training data pool Xr,, budget b, pretrained aesthetic scorer Saes concept-matching scorer Scon (-), adaptor A, learning rate group R.
Input:
Output: Updated training pool Xr, updated adaptor A.
1: Fine-tune the aesthetic scorer Saeel-, 0) on XT following Sec
B.3
2: Calculate the aesthetic score Yaea (A)
[ga(Xr>) Exg Saea (2g) and concept-matching
score Ycon (A) [=a(Xx]] Ln, Eaa (XT) Scon (Igg) for A:
3: for x Dpoot do
4: Calculate the integrated score ((x)) 7
5: end for
6: B (Top-b samples within Dooo according S);
7: s +- -AA( (Eq,());;
8: <- the largest value within R:
9: while r + min(R) do
10: A' < -V Le LLDM (a, A);
11: Generate small-scale XG with g(-) conditioned on A';
12:
Calculate score ((A) for A via () and Xa:
13:
if 7A) then
14:
A + A';
s (A);
16:
else
r +- smaller value than r within R
18:
end if
end while
20:
X, < XB
We use weighted-scoring system as the acquisition module that evaluates the quality of images and adaptors in our proposed CATOD framework. In traditional active learning or data selection, the acquisition module only conducted on samples. However, we observed that an increase the number of samples does not necessarily contribute to performance boost like in traditional active learning setting (which we will verify in Appendix C.. Therefore, we propose this weighted scoring system to help schedule adaptor training, which helps locate the best version among them and select samples according to their quality.
Concepts framework can be divided into 4 steps: (1) Train the scoring system on the current training set XT. (2) Train the adaptor on the training set XT and schedule the training accordingly to the scoring system; (3) Calculate the acquisition score accordingly to the scoring system and the best adaptor for samples in dataset Dpoot: (4) Select top-K samples from Droot and move them to train set Xy.
Our Controllable Adaptor Towards
B.2 An Implementation of MMD score.
To give out an quantifiable and unbiased estimation of Eq. we sample two sets of vectors X = {I1, I2,,.., Fm from P and G = {91. 92, ...,.... from Q, an estimation can be give by:
m
m
m
2 m n
d MD (X, m(m 1)
k(r,, 1.)+ n(n 1)
klg, (i)
mn
#
(17)
B.3 Aesthetic Scorer
Following the description of the aesthetic scorer in Section 3.2 we explain how to personalize it here. Since our primary goal to make the model distinguish low-qualit/high--qaiiit examples for OOD concepts, we design automatic procedure to assign aesthetic scores to our training set X at each cycle, based on which we attain an aesthetic training set Xaes. In this work, the aesthetics training comprises three parts: (1) The original training set Xy; (2) Generated set DrC without assistance
19

---

Aesthetic
Samples
Latent
High Aesthetic
Variable
Te(e)
High Quality
Conepx-Matching
Samples
High Concept-Matching
Dynamic Schedale
Idcal Image Manifold
Fival Schalole madc-ul))
Figure 6: An intuitive comparison for fixed/dynamic schedules. The active learning paradigm can be viewed as guiding the iterative embedding updating through newly added samples. We can see that a fixed schedule makes the learned embedding heavily biased, which in turn leads to performance fluctuation.
from any adaptors: (3) Randomly sampled set Dsr from similar but irrelevant categories. Since we focus on the concepts that the text-to-image model can hardly recognize and visualize, DTG contains lots of plausible, disrupted, or even irrelevant samples, which we assign a zero score. This helps the aesthetic model understand that originally generated samples are low-quality samples. In contrast, Dsi somehow describes some attributes for the concept but still does not match the concept, to which we assign the average score of 5.0. We employ the normal regression loss formulated as follows:
Lues
1
Σ E,EXT
Ilp..Pell
Σ
1I - -P.l.
(18)
X
DDG + Der
d'EDroUDsr
where p denotes score predicted by the original scorer, when denotes the score we assign, and is set to 1.0 in our experiments. Following this paradigm, we train an aesthetic scorer for each category of concepts, and the score personalized for each category from Ppoot can be predicted accordingly.
After selecting top-K data according to this score based on adaptor quality, we assign a lowered aesthetic score to them:
P2 10.0 tearrent (10.0 Pw),
(19)
total
where Ps denotes the score given by generic aesthetic scorer, total represents the number of total active learning cycles, when feurrent indicates the current cycle. Note that the newly selected images typically have lower quality than those from earlier cycles, assign progressively lowered scores to them as the learning cycle proceeds.
B.4 Adaptor Evaluation and Schedules
Following the content in the second part "The Active Schedule for Training Adaptors" in Sec. 3.3 we describe our schedule in more detail. In training CATOD, we use this indicator as a signaling proxy throughout the training process. As the number of training samples increases, the quality of selected samples later cycles might lower, and fixed schedule may introduce some unnecessary details within these images (such watermarks, borders, disturbing objects, etc.). However, these subsequent samples with relatively lower quality do contain some good photographic attributes that can help the embedding evolve. Actually, the potential problems can be alleviated by carefully-designe schedules. An intuitive illustration for this schedule is given in Fig. 6
We first train the adaptor for 5 epochs, and save an adaptor version A; per 5 epochs, together with its corresponding generated images, constituting sub-cycle. When this sub-cycle is completed, we filter out the adaptor with the highest quality based on the score (A) (6).(6). ). If the selected adaptor has the most training epochs across versions, we keep the learning rate and conduct next sub-cycle, otherwise, we will reduce the learning rate. Note that if the adaptor quality of this cycle not as good as that of the previous cycle (because of the quality degradation caused by introducing some unnecessary details), the initial learning rate will be reduced, which helps avoid
20

---

Table The statistics of the dataset we use. The test data size #D of all the concepts is fixed to 100 for a fair comparison, while the training data size #D varies since different concepts since the number of data samples across publicly available datasets different.
Iracel
CMMD 42
Lixand
CMMD ())
Peaguis
CMMD G3
Scalih
CMMD (1)
Stake
CMMDD(
Aniiun
Empener Penpain
Crmptiih
neate
Frilled
Poraitic
Mediterazcinn House Gecko
Gartah
Indion
Xylonczhux
4
Ocdara
Buckhapper Perguis
4.0c
Tiaafah
4,17
Pelanta Plaurus
3.94
Zopherinse
12
Optandie
3.88
Royal Pengui
3.98
Ture
4.0
Skfrorader
4.11
Theps
Whigtal
Empene Pengrin
Gelate
Thander state
1.45
Hannesheul
Aptrads
Night
Red
Pesguin
Shack
Prtson
1.76
Meadin Malh
Kirmoda Drigon
Addlic Penguen
Kilke Whale
Hegnue Stake
1.94
Table 6: The statistics of the dataset we use. The test data size #D of all the concepts is fixed to 100 for fair comparison, while the training data size #Dpool varies since different concepts since the number of data samples across publicly available datasets is different.
Category
Concept
# D Ppoot
#Deat
Zopherinae
1357
100
Antlion
864
100
Insect
Lycorma Delicatula
5108
100
Parasitic Wasps
877
100
Xylotrechus
1043
100
Axolotl
1200
100
Frilled Lizard Mediterranean House Gecko
1008
100
Lizard
889
100
Oedura
798
100
Opluridac
818
100
Emperor Penguin Chick
987
100
Gentoo Penguin
3992
100
Penguin
King Penguin Chick
1471
100
Rock Hopper Penguin
877
100
Royal Penguin
754
100
Crampfish
1179
100
Dragonfish
835
100
Sea Fish
Garfish
1200
100
Tigerfish
538
100
Tuna
1236
100
Ahaetulla Nasutar
893
100
Aipysurus Laevis
1015
100
Snake
Indian Cobra
1200
100
Pelamis Platurus
1091
100
Sidewinder
1200
100
introducing disturbing elements led by some samples with insufficient quality. The sub-cycles will be continuously conducted until the minimum learning rate is reached or the adaptor quality no longer increases. Our learning rates include 5 x 10 2,-5 2.5 x 10 7.5 10 10-3,5 x 10-5 5,55 x 10-5,
2.5
C Experiments
C.1 How to Automatically Locate OOD Concepts
Since the publicly available dataset, i.e., ImageNet, iNaturalist 2018 [67), IP102 (71), which we adopt in our research contains 9,244 concepts with 14,883,455 images in total, it is infeasible to manually test and decide what concept is the out-of-distribution concept one by one. Therefore, our procedure for figuring out is two-fold: (1) automatically quantifying the distribution drift of the given concept from what the text-to-image model (Stable Diffusion 2.0, a.k.a a SD 2.0) learned with
21

---

the CMMD metric (thanks to the recent work proposed by Google (27]): whether the text-to-image model could generate the right content, according the rank given by the CMMD metric above. Specifically, in the first step, we randomly sample 10 image data for each concept within the datasets and then generate 10 images with the model with the text prompt "A photo of St", in which 5 denotes the concept name. Then, we calculate the CMMD discrepancy between these two data batches and record them. The CMMD scores of some concepts (including both Out-of-Distributie concepts in the first 5 rows and In- In-Distribution in the last 5 rows) are listed in Table
(2) Manually verifying
Empirically, we observed that SD 2.0 is unable to synthesize the concepts with a CMMD metric above 3.5. Therefore, we pick 25 OOD concepts accordingly divide them into categories, and make our dataset as follows:
Insect: Zopherinae, Antlion, Lycorma Delicatula, Parasitic Wasps, Xylotrechus;
Lizard: Axoloti, Frilled Lizard, Mediterranean House Gecko, Oedura, Opluridae;
Penguin: Emperor Penguin Chick, Gentoo Penguin, King Penguin Chick, Rock Hopper Penguin, Royal Penguin;
Sea Fish: Crampfish, Dragonfish, Garfish, Tigerfish, Tuna;
Snake: Ahaetulla Nasutar, Aipysurus Laevis, Indian Cobra, Pelamis Platurus, Sidewinder.
Actually, the dataset we collect is not baass-balance,, so the number of samples we can collect varies across different concepts. The statistics of each concept are shown in Table For the concepts with a D pooo less than 1,000 images, we also collect some of them from image host websites including Dreamstime, Flicker, istockphoto, pinterest, and shutterstock. The dataset will be released as soon as the code is released.
C.2 Experimental Settings
To verify the effectiveness and extensibility of our proposed CATOD framework, we conduct experi- ments on 25 different OOD concepts that the large-scale text-to-image (i.e. Stable Diffusion 2.0 (45)) implement different versions of CATOD. This section explains some important implementation details for the dataset, the aesthetic scorer, the concept-matching scorer we use, and how we design the learning paradigm.
The Learning Rate Schedule. As shown Algorithm 1 we have a learning rate group R, which includes 5 learning rates; 5 x 10-4, 2.5 x 10-4, 7.5
10 10 5, 5 x 10 2.5 10 Following the indicator ((4) in Eq. (6). we calculate this value after every epoch. When this indicator falls below the previous evaluation,
reduce the learning rate. If the lcarning rate cannot be lowered further, we conclude that the model has converged and the training is complete. Note that we choose to train 20 epochs in all experiments as claimed in the main context, it is possible that the training is early stopped before reaching epoch 20.
Aesthetic Scorers. To offer the basic knowledge for aesthetic evaluation, we use pre-trained generic models, including ReLIC [8T), TANet [23), SDDChad Scorer, and VILD (59). Note that the scores given by the generic aesthetic scoring model tend lie in the majority score range (5 to 6) in our dataset, we ought to personalize this model accordingly to our training set, which we refer to as PIAA [4 Since PIAA typical small sample problem, we adopt similar experimental settings and evaluation criteria by referring to Few-Shot Learning [16] and a previous PIAA research work: PA-IAA 33
Concept-Matching Scorers. Since the first step of general image retrieval is feature extraction, we use the CLIP ViT-L/14 encoder, which is commonly used and reliable feature extractor
C.3 About Compositional Results over Multiple-Concepts.
To further validate the effectiveness of CATOD, we also present compositional results based on the LoRA adaptors we obtained, as illustrated in Figure [ We compose different types of concepts, including (i) background concepts such as brick wall and grassland for Emperor Penguin Chicks,
22

---

Ground Truth
with an Alligator Lizard
with an Opluridac
+ on desk
in the beach
+ with dog
Frilled Lizard
Ground Truth
+ against brick wall
+ in the grassland
+ with an Adelic Penguin
with an King Penguin Chick
with dog
Emperor
Penguin
Chick
Figure 7: Generative results with 2 concepts within one image. Experiments are conducted based on the LoRA adaptor fully trained on concepts "Frilled Lizard" and "Emperor Penguin Chick". We try to compose these creatures with background elements, in-distribution concepts, and out-of-dittriutiion concepts learned by other adaptors. The final results show high quality with minimal disruptive details.
and desk and beach for Frilled Lizard; (ii) in-distribution and common concepts, like the dog; (iii) in-distribution and similar concepts, like Adelie penguin for Emperor Penguin Chicks, and Alligator Lizard for Frilled Lizard; (iii) out-of-distribution concepts from other adaptors, like King Penguin Chicks for Emperor Penguin Chicks and Opluridae for Frilled Lizard. Our observations reveal that the synthetic results on out-of-distribution concepts with CATOD can seamlessly integrate different background elements, even if they were not present in real-world images (columns to 2). For other in distribbutiin concepts like dogs, LDMs tend represent a specific species with similar color and texture, while different concepts remain highly distinguishable within one image (columns 3 to 4). For out-o--distribution concepts, we also note that the creatures are depicted correctly, but may exhibit some confused visual details that negatively impact the aesthetics of the image (column 5).
C.4 More Analysis.
Eyes
(a) The number of epochs with specified different learning rates within the schedule RAND.
(b) The number of epochs with specified different learning rates within the schedule CATOD.
(c) Changes on different scores as cycle proceeds on RAND.
(d) Changes on different scores as the cycle pro- ceeds on CATOD.
Figure 8: A comparison on how the schedule and scores change on RAND(scheduled) and CATOD as eycle proceeds on concept emperor penguin(chick). (a).(b) show how the #epochs for each learning rate in the schedule change as the cycle proceeds, when (c),(d) show how score change on RAND (scheduled) and CATOD. The scores for CATOD stop changing at cycle 12 since more added samples do not help boost adaptor quality.
CATOD is more reliable on schedules. Figure 8 (b). show comparison of how our schedules adjust as the number of cycles increases on (cheusch)d and CATOD. Both of them show the tendency that training epochs with larger learning rates decrease when those with small learning rates
23

---

Shape
Shape 2
Shape
Ground Truth
TT-AND(FaOo)
TH-AND
Fieed
CATOD
90
00
Gencrated Images
No. Cydo
(a) R-precision( %) with different strategies as training pool expands. The experiments are per- formed over concept "emperor penguin(chick)" and produced show the impact of number samples, we also compare their modified versions that do not use dynamic schedules.
(b) A comparison on real-images with CATOD- generated ones in different shapes of lycorma delicatula. We can see that with carefully de- signed selection, the adaptor helps produce all the shapes the concept "lycorma delicatula".
RAND and CATOD. To further
Table 7: A Comparison over the performance of CATOD different types of the initial training data pool, in terms of the CLIP score CMMD score with 100 images sampled at last. This table shows the average result of 5 sub-elasses within each category. The overall improvement our proposed CATOD provided by "Imp.". Methods with the best performance are bold-folded.
CLIP Score (?) scafish
CMMD 27 (L)
Comparison Methods
insect
lizard
ponguin
snake
Avg.
lizard
penguin
scafish
snake
Avg.
10 HQ initial samples
73.87
79.23
75.59
83.84
82.19
78.94
0.87
0.80
0.71
0.79
0.78
0.79
RAND initial samples
72.26
74.06
72.89
79.75
78.08
75.41
0.92
0.84
0.73
0.80
0.93
0.84
20 HQ initial samples
75.18
77.59
76.97
82.78
83.14
79.13
0.85
0.79
0.69
0.72
0.75
0.76
RAND initial samples
72.08
72.82
72.97
77.74
75.06
74.13
1.01
1,03
0.92
0.97
0.83
0.95
50 HQ initial samples
75.27
77.27
76.85
82.90
83.37
79.13
0.84
0.80
0.63
0.69
0.71
0.73
RAND initial sample
70.34
70.53
67.49
72.10
73.74
70.84
1.13
1.29
0.94
1.06
0.97
1.08
increase, meaning that the training schedule focuses more on fine-tuning later cycles. Comparing CATOD with RAND, we can see that the epochs with the maximal learning rate already diminish at cycle on CATOD while RAND still needs large learning rates, indicating that CATOD are better at recognizing the given concept and tends to be more stable.
CATOD achieves good score earlier than other methods. Figure Figure(Scc),cll shows comparison of how the httpi///ooncppmmmmcccinommppeppe and CATOD. We see that CATOD already achieves higher value all scores and cannot be further boosted with 120 training samples on cycle 12 when the performance for RAND still fluctuates. This phenomenon illustrates that active selection strategy makes our embedding be trained on more high-quality data.
scores change as the cycle proceeds on RAND
Furthermore, also notice that aesthetic/concept-maiie the comprehensive score is enhanced continuously, which exactly corresponds to our proposed dual scoring system that balances the importance of these two factors when ensuring the quality of the adaptor never declines.
scores fluctuate later cycles when
The performance of CATOD on different initial training data pool. We also ablate CATOD over the initial training set size/quality. detail, we retest CATOD with different numbers of initial samples (10,2,,50) and different quality, i.e., high-quality (HQ) and random sampling (RAND), with 100 images sampled last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results shown in Table 7
24

---

Table 8: A Comparison the performance of CATOD different architectures, in terms of the CLIP CMMD score with 100 images sampled last. This table shows average result of sub-classes within each category. The overall improvement of our proposed CATOD is provided by "Imp.". Methods with the best performance are bold-folded.
Compasison Methods
insect
lizard
penguin
CLIP Score (1)
scatish
stake
Avg-
Imp.
insect
lizard
penguin
CMMD seafish
27 (
snake
Avg.
Imp.
RAND
66.29
69.45
66.07
1.49
1.33
(0.61
1.5 + CLIP
68.78
70.38
70.89
74.20
73.89
71.63
13.71
1.19
1:45
1.28
1.45
1.21
1.32
(0.12
SD 1.5 + CATOD
71.73
74.79
73.45
79.37
77.35
75.34
1.01
0.97
0.78
0.92
0.84
0.90
SDXL RAND
72.39
73.04
71.75
74.81
70.47
72.49
18.05
1.39
1445
1.49
1.58
1.01
1.38
100.50
SDXL CLIP
79.32
78.24
79:75
78
11.05
1.16
0.93
1.07
1.30
J0.22
SDXL CATOD
80.37
79.58
77.56
85.20
79.97
80.54
0.95
0.87
0.96
0.89
0.75
0.88
Table 9: A Comparison over the diversity score of CATOD, in terms of the CLIP score and CMMD with 100 images sampled last. This table shows the average result of sub-classes within each category. The overall improvement of our proposed CATOD provided by "Imp.". Methods with the best performance are bold-folded.
LPIPS Score(f) seafish
Comparison Methods
insect
lizard
penguin
snake
Avg.
Imp.
DreamBonth + CLIIP
0.254
0.156
0.149
0.305
0.265
0.226
10.052
DreamBooth + CATOD
0.362
0.198
0.197
0.349
0.286
0.278
TI CLIP TI + CATOD
0.198
0.257
0.174
0.108
0.208
0.189
110.333
0.267
0.278
0.152
0.278
0.133
0.222
LoRA CLIP LoRA CATOD
0.178
0.184
0.155
0.203
0.094
0.163
110.054
0.245
0.203
0.217
0.244
0.178
0.217
From these results, we can draw the following conclusions:
Initial batch size has a more significant impact on randomly initialized samples than on high-quality samples. Specifically, the performance change with high-quality samples is at most 1.40 in the CLIP score and 0.10 in the CMMD score concerning different initial numbers of training data. In contrast, the performance change on low-quality up 7.65 in the CLIP score and 0.45 in the CMMD score.
The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends be even more significant.
C.5 Further Extension: Concept with multiple shapes
Recall that Stable Diffusion 2.0 fails to generate emperor penguin chicks, which accounts for the fact that adult emperor penguin chicks and their chicks have different appearances. We raise another question: can we train an adaptor with a concept that has different shapes? Inspired by this, we also test our proposed CATOD on the concept "lycorma delicatula",, which is kind of pest with 3 different forms its life-cycle, and surprisingly find that with 300 training samples, the best version of adaptors successfully produces all these 3 shapes, shown in Figure (b)) These results further validate the effectiveness of our framework, even the presence of appearance ambiguity.
C.6 A bout Experimental Results on other Architectures
Our experiments in the main content are all conducted on Stable Diffusion 2.0 (SD 2.0). To further validate the superiority of our proposed CATOD, conduct additional experiments other architectures (based on LoRA), including Stable Diffusion 1.5 (SD 1.5) and Stable Diffusion XL. (SDXL). The results shown in Table
This table shows that our proposed CATOD also compatible with different architectures with notable performance gain compared to the basclines.
25

---

Selection
Generation
LoRA + CLIP
Axolotl
LoRA CATOD
LoRA + CLIP
Emperor
Penguin
Chick
LoRA CATOD
Figure 9: A comparison of selected and generate samples on different combinations of methods and concepts. We can observe that training samples with different angles selected by CATOD also lead to diverse angle in their generative results.
C.7 About the diversity
CATOD maintains the diversity to produce OOD concepts with different angles poses in generative results. To validate the diversity of our generative results, we first provide quantitative evaluation with LPIPS 80, 78 The results are shown in Table In this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, conclude that CATOD also preserves diversity, Since the training set contains samples with different angles in CATOD, it also easy to produce objects with different angles, and show the examples on concepts "Axolotl" and "Emperor Penguin Chick" in Figure 9 We can observe that training samples with different angles selected by CATOD also lead to diverse angle in their generative results.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly summarized our contribution in the abstract and in the last paragraph of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
We believe that the limitation should not be regarded as an integral part of this paper.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiittinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
27

---

Answer: [Yes]
Justification:
We have provided theoretical proof of our theory in Sec 4 in Appendix. A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided code within the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: This paper does not provide the dataset regarding the risk of anonymity, despite that the dataset is just collection of publicly available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.ccpublle wwwppppgpuuiiieeeee /esic///uuide//oeeuuuiiiiiii SooodeSubiissi.e ssiiiioliccc for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All experimental details are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Our experiments do not have error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is reported in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpps//neurips. httprrpss...//uloleccccccccceeee
Answer: [Yes]
Justification: Our paper has no risk regarding ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This has been briefly discussed our conclusion section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: MIT License has been added already.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have readme document for our codes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments were conducted.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No crowdsourcing experiments were conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "FhoQYlcQxc", "reviewer": "Reviewer_yLUC", "summary": "This work presents CATOD, a new method for dealing with the challenging scenario of adapting generative models to OOD concepts. In particular, CATOD leverages an active learning setting to update the adaptor to generate better OOD images more broadly. Furthermore, the authors provide both extensive empirical evaluations and theoretical analysis which further support the validity of the CATOD methodology.", "questions": "The reviewer would like to have further clarification regarding the experimental setup of the paper. In particular, the reviewer would like to hear the author more directly address whether the small evaluation set used for the experiments is a reasonable concern.", "limitations": "The authors don't provide any full discussion of the limitations of the proposed methodology. Additionally, no direct comment has been made regarding societal impacts, however, the reviewer does not see any potential negative impacts derived from this work.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Strength:\n- This paper provides a timely analysis of the difficult problem of adapting generative models to OOD concepts which is often overlooked in recent literature.\n- CATOD showcases both strong empirical performance (CLIP score and CMMD score) on a reasonable set of 25 OOD concepts.\n- Additionally, the paper provides strong theoretical links between the introduced aesthetic/concept-matching score and their necessity to the performance of adaptors.", "weaknesses": "The primary concern with this paper is the sparse empirical evaluations presented for CATOD. In particular, the 25 OOD concepts chosen seem few with only a small set of 100 samples left out for validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments!\n\n**Part 1: About the number of samples to use for training and validation.**\n\nIn brief, since most recent adaptors require only a small number of training samples (usually around 100), our validation set is also set to be of the same scale as the training samples.\n\nAs mentioned in the paper, *adaptors* are designed to reduce training costs when introducing new concepts. Specifically, they require significantly fewer samples compared to direct fine-tuning, typically no more than 1000 samples, and often around 100 samples. The following table provides a comparison of the most commonly used adaptors (including our proposed CATOD):\n\n| Aspect                       | Textual Inversion [1]                                     | DreamBooth [2]                                             | LoRA [3,4]                                                   | CATOD (Ours)                                             |\n| ---------------------------- | --------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------ | -------------------------------------------------------- |\n| **Training Data**            | Requires a few images (5-20)                              | Requires a large amount of labeled data (100-1000)         | Requires intermediate amount of data (around 100)            | Requires intermediate amount of data (around 100)        |\n| **Flexibility**              | High flexibility, but usually limited to only one concept | Not flexible, can learn multiple detailed attributes       | High flexibility, can learn multiple detailed attributes     | High flexibility, can learn multiple detailed attributes |\n| **Model Size**               | No change to model size                                   | Model size can increase                                    | Slight increase due to additional adaptation layers          | Depend on the adaptor, usually slight increalowse        |\n| **Quality for New Concepts** | Usually not good                                          | Good for ID concepts, but usually deteriorates on OOD ones. | Good for both ID concepts, but usually deteriorates on OOD ones. | Good for both ID and OOD concepts.                       |\n\nSince previous adaptors often fail to accurately depict out-of-distribution (OOD) concepts due to inconsistent image quality, we also ensure that our validation set is of high quality to provide accurate validation.\n\n[1] Gal, Rinon, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. \"An image is worth one word: Personalizing text-to-image generation using textual inversion.\" *arXiv preprint arXiv:2208.01618* (2022).\n\n[2] Ruiz, Nataniel, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. \"Dreambooth: Fine-tuning text-to-image diffusion models for subject-driven generation.\" In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 22500-22510. 2023.\n\n[3] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" *arXiv preprint arXiv:2106.09685* (2021).\n\n[4] Yeh, Shih-Ying, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. \"Navigating text-to-image customization: From lycoris fine-tuning to model evaluation.\" In *The Twelfth International Conference on Learning Representations*. 2023.\n\n\n\n**Part 2: About the number of categories to use for the adaptors.**\n\nThank you for reminding me to investigate the capacity of adaptors in learning categories with OOD concepts! Extending CATOD to encompass even more categories is straightforward. Using the category \"Insect\" as an example, we explored the impact of adding three additional OOD concepts. Specifically, we conducted experiments with the following additional categories: \"Chlumetia transversa,\" \"Mango flat beak leafhopper,\" and \"Rhytidodera bowrinii white.\" The table below shows the average performance in the \"Insect\" category as we add new concepts:\n\n| Number of Categories            | DreamBooth + CATOD (CLIP $\\uparrow$) | DreamBooth +for CATOD (CMMD$\\downarrow$) | LoRA + CATOD (CLIP$\\uparrow$) | LoRA + CATOD (CMMD$\\downarrow$) |\n| ------------------------------- | ------------------------------------ | ---------------------------------------- | ----------------------------- | ------------------------------- |\n| 5                               | 70.83                                | 1.39                                     | 71.19                         | 1.13                            |\n| 6 (+Chlumetia transversa)       | 69.23                                | 1.57                                     | 68.05                         | 1.52                            |\n| 7 (+Mango flat beak leafhopper) | 66.35                                | 1.96                                     | 65.39                         | 2.03                            |\n| 8 (+Rhytidodera bowrinii white) | 64.07                                | 2.65                                     | 63.28                         | 2.77                            |\n\nFrom this table, we can see that while CATOD can be easily generalized to more concepts, it experiences a performance decline as the number of categories increases. This decline is because recently proposed adaptors were originally designed to accommodate only a few concepts. However, since CATOD is designed as a general framework, it can be easily adapted to work with any future adaptors that are capable of handling a larger number of concepts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UGAWSqXaOA", "reviewer": "Reviewer_PRf3", "summary": "This work tackles the challenge of adapting text-to-image diffusion models to out-of-distribution (OOD) concepts. The authors introduce a framework called Controllable Adaptor Towards Out-of-Distribution Concepts (CATOD), which employs an active learning paradigm to iteratively accumulate high-quality training data and optimize adaptor training. The framework features a dual scoring system that balances aesthetics and concept matching, ensuring improved quality and conciseness of the generated images.", "questions": "1. Is the CMMD shown in Figure 1 calculated using Real Images and before adapting images?\n2. Are the high-quality samples shown in Figure 2 generated by vanilla Textual Inversion, DreamBooth, and LoRA? Or are they already using the CATOD framework?\n3. Is “if r_k is the closet representation for r_k” in line 145 of the paper correct?\n4. Eq. 6 and the description in lines 173 to 174 of the paper confuse me. Shouldn’t the maximum value of \\gamma (A) be \\gamma_aes (A)=\\gamma_con (A)=10+40k, k=0...n, and the minimum value be \\gamma_aes (A)=30+40k, or \\gamma_con (A)=30+40k, k=0...n?\n5. Associating the two terms of Eq. 10 directly with aesthetic and concept-matching scores requires further theoretical explanation and empirical analysis.\n6. I noticed in the experiment section that the training period is about 20 epochs. Would this somehow be too less? If there were enough training cycles, we would get an overfitted LDM capable of generating images containing specific OOD concepts.\n7. In Algorithm 1, lines 9 to 19, it does not look like new training samples are being selected for each epoch.\n8. Does the learning rate set R determine the stopping of the training? How would you guarantee that the model has converged at this point? As question 5, a sufficiently overfitted LDM should be able to generate these OOD concepts.\n9. This paper tests the proposed CATOD on SD 2.0. Have you conducted experiments to evaluate its performance on other text-to-image models, such as SD 1.5 and SDXL? Including these results would help verify the versatility of your method.\n10. I am curious about how CATOD performs with different initial training set sizes (e.g., 10 and 50) and varying quality. Adding this as an ablation experiment would provide valuable insights into the robustness of your method.\n11. I am also interested in whether similar categories (e.g., other kinds of penguins) might be distorted after adaptation. Have you conducted any experiments to evaluate the degree of distortion in these cases?\n12. Since CMMD is a newly proposed metric by Google in CVPR 2024, it would be helpful to list the advantages of using CMMD compared to traditional metrics like Inception Score, FID, and MMD. This would help readers understand the rationale behind choosing CMMD.", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper is easy to follow due to its clear writing, concise figures, and well-structured formulation of problem definitions, methods, and theories.\n2. The motivation of this paper is innovative, addressing the overlooked problem of adapting recent text-to-image (T2I) models to out-of-distribution concepts.\n3. The paper presents sufficient experimental results that effectively support its claims.", "weaknesses": "1.\tThe key contribution of this paper is Equation (6) and (7), which are claimed to be use as a signal to reduce the learning rate, stop the training in time, and select samples for the next cycle.  It seems the innovation is that the authors identify two relevant factors and utilize the strategy of active learning. Am I correct? \n2.\tAuthors use MMSE to reinterpret the loss function of the LDM and decompose it into two terms (Eq. 10). As the description in lines 213 to 220 of the paper, associating these two terms directly with aesthetic preservation and concept-matching score is rather far-fetched and vague. Kindly advise if I am wrong.\n3.\tIf the aesthetic and concept-matching scores are really that important, why just use those two scores to pick suitable samples for adaptation from the beginning? This would also avoid the later calculations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We have carefully addressed your concerns as follows:\n\n**Part 1: About the key technical contribution (W1)**\n\nThe key technical contributions of this paper are twofold: (1) our method for estimating the impact of each sample on the model without training, corresponding to the aesthetic/concept-matching scorer described in Section 3.2; (2) our approach to effectively utilizing these scores and balancing the trade-off between the two impact factors (Eq. 6 and Eq. 7). This is achieved through Active Learning, a vital paradigm that enables dynamic trade-offs cycle-by-cycle.\n\n**Part 2: A more detailed explanation for the theories (W2, Q5)**\n\nWe are pleased to provide a detailed explanation of the connection between our proposed method and the underlying theories!\n\nAs introduced in our background Section (Sec.2), the LDMs consist of two core components: An encoder $\\mathcal{E}$ learns to map images $x$ to a latent code $z=\\mathcal{E}(x)$, and a diffusion model trained to produce code conditioned on texts. Let $c\\_\\theta(y)$ be a model mapping the conditioning text $y$ into latent vectors, $A$ the adapter embedded into the diffusion model, the LDM loss is given by:\n$L\\_{LDM}(x,A)= \\mathbb{E}\\_z\\sim\\mathcal{E}(x),t,\\epsilon\\sim\\mathcal{N}(0,1)\\left[\\Vert \\epsilon-\\epsilon\\_{\\theta, A}(z\\_t,t,c\\_{\\theta, A}(y)) \\Vert\\_2\\^2\\right]$ in Eq.1, where t is the time step, $z\\_t$ the latent noised to time $t$, $\\epsilon$ the noise sample, $\\epsilon\\_\\theta$ the denoising network. When optimizing the LDM loss, there are two important factors we should consider: the adapter $A$ that adapts the underly model to OOD image $x$, and the conditional text $y$ for the images $x$ to match. As we claimed in the main context that disrupting $x$'s can lead to misunderstanding of concept $y$, our objective is in Eq.2:\n\n$A^*,X\\_{T}^*=\\mbox{argmin}\\_{A,X\\_T} L\\_{\\mbox{LDM}}(x,A,y)$. \n\nSince the optimal set for $A, X_T$ is initially unknown and cannot located at once, is common to use an iterative paradigm that alternatively optimizes $A$ and $X_T$. We have the following conjugate forms:\n\n$\\mathbf{X}\\_T\\^{(t)} = \\mbox{argmin}\\_{\\mathbf{X}\\_T\\subset D\\_{pool}}\\mathbb{E}\\_{x\\sim \\mathbf{X}\\_T}L\\_{LDM}(x,A\\^{(t-1)},y)$ (Eq.3 in main context)\n\n$A\\^{(t)} = \\mbox{argmin}\\_{A}\\mathbb{E}\\_{x\\sim \\mathbf{X}\\_T\\^{(t-1)}}L\\_{LDM}(x,A,y)$ (Eq.4 in main context)\n\nFor the first equation (Eq.3), we optimize the training $X_T$ towards both conditional text $y$ (image-text matching) and loss reduction (aesthetic preserving), which in turn leads to the motivation of decoupling aesthetic/concept-matching scores. Theorem 4.3 verifies the necessity of this decomposition while Section 3.2 implements this decomposition. The second equation (Eq.4) just performs an ordinary adaptation on select samples.\n\nSince the iterative paradigm in Eq. 3 and Eq. 4 may lead to convergence to local optima, we also need a dynamic weighted scoring system to trade-off between aesthetics and concept-matching, to alleviate the potential bias towards only one of them (leading to the practical implementation in Sec 3.3). To this end, we have explained the motivation of our theory and how we organize our CATOD.\n\n**Part 3: The pre-calculated scores are not feasible (W3)**\n\nSince Out-of-Distribution (OOD) concepts are usually unseen by both the underlying text-to-model and the aesthetic/concept-matching scorers, it is likely that the preset scorers provide inaccurate results. To address this, we must use the currently available high-quality samples to correct these inaccuracies. Following the iterative paradigm outlined in Part 2, the scorers and adaptors are alternately adjusted to ensure accurate results. Additionally, we update the scorers after selecting high-quality samples in each cycle. To demonstrate this experimentally, we conduct another ablation experiment with pre-calculated scores, as shown in the following table:\n\n| Method                           | Lizard (CLIP$\\uparrow$) | penguin(CLIP$\\uparrow$) | Lizard (CMMD$\\downarrow$) | penguin(CMMD$\\downarrow$) |\n| -------------------------------- | ----------------------- | ----------------------- | ------------------------- | ------------------------- |\n| CATOD                            | 77.00                   | 74.11                   | 0.89                      | 0.71                      |\n| CATOD with pre-calculated scores | 75.29                   | 73.57                   | 0.93                      | 0.76                      |\n\nWe observe a noticeable performance loss when the scores are pre-calculated.\n\n**Part 4: Other architectures (Q9).** \n\nThanks for the advice! It is straightforward to extend our experiments to other text-to-image models (like SD 1.5 and SDXL). We show some of these results with LoRA in the tables below:\n\n| T2I Model Ablation        | Axolotl (CLIP) | Axolotl  (CMMD) | Emperor Penguin Chick (CLIP) | Emperor Penguin Chick (CMMD) |\n| ------------------------- | -------------- | --------------- | ---------------------------- | ---------------------------- |\n| CATOD + SD 2.0 (original) | 75.35          | 0.85            | 74.89                        | 0.79                         |\n| CATOD + SD 1.5            | 74.79          | 0.88            | 73.01                        | 0.92                         |\n| CATOD + SDXL              | 79.36          | 0.72            | 80.37                        | 0.66                         |\n\nThis table shows that our proposed CATOD is also compatible with different text-to-image encoders with notable performance. We will add these ablations to our final revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G7R4lvYljG", "reviewer": "Reviewer_bHo6", "summary": "This paper introduces a novel framework called Controllable Adaptor Towards Out-of-Distribution (OOD) Concepts (CATOD). CATOD is designed to adapt text-to-image models to OOD concepts and generate high-quality images of those OOD concepts accordingly. The authors identified the challenge of accurately depicting OOD concepts due to low-quality training data. CATOD employs an active learning approach to iteratively accumulate high-quality training data and update the adaptor. The framework incorporates an aesthetic score and a concept-matching score to guide the accumulation of training data. Extensive experiments demonstrate significant improvements in both the CLIP score and the CMMD metric.", "questions": "1. Following weakness 1, can you compare the diversity of the fine-tuned model under different strategies?\n2. It would be intriguing to investigate how initial data size impact the experiments, especially for the multiple concept scenario as you mentioned in Sec. 5.2 following weakness 2.\n3. For different concepts, which of the two scores gets high weightage (in equation 11) as the training progresses?", "limitations": "See weakness.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation of the paper is clear and concise. The method is easy to understand and the delivery of the paper is smooth.\n2. The paper addresses a significant challenge in text-to-image generation by focusing on OOD concepts, and serves as the first work to address the OOD challenge from a data-centric perspective.\n3. The method is well-motivated and supported by theoretical insights into the importance of aesthetics and concept-matching scores.\n4. The experimental results demonstrate significant improvements over existing methods.", "weaknesses": "1. To support the claim in Line 189 that your selection \"guarantees the sample diversity\", the authors should report the diversity metrics of the generated samples. \n2. The paper provides ablation results on the design of weighted scoring strategies, which is valuable. I think more extensive ablation studies on the impact of initial data size, the type of text-to-image encoders, etc. can help signify the paper's contribution.\n3. To further investigate how the weighting strategy work in your proposed CATOD, it would better to list how the weights of aesthetic/concept-matching scores change as your learning cycle proceeds over different concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions! We hope that our rebuttal addresses your concerns.\n\n**Part 1: About the diversity evaluation (W1, Q1)**\n\nCATOD maintains the diversity to produce OOD concepts with different angles or poses in generative results. To validate the diversity of our generative results, we further provide a quantitative evaluation with LPIPS~[1,2]. The results are shown in the Table below:\n\n| Comparison Methods | Insect    | Lizard    | Penguin   | Seafish   | Snake     |\n| ------------------ | --------- | --------- | --------- | --------- | --------- |\n| DreamBooth + CLIP  | 0.254     | 0.156     | 0.149     | 0.305     | 0.265     |\n| DreamBooth + CATOD | **0.362** | **0.198** | **0.197** | **0.349** | **0.286** |\n| TI + CLIP          | 0.198     | 0.257     | **0.174** | 0.108     | **0.208** |\n| TI + CATOD         | **0.267** | **0.278** | 0.152     | **0.278** | 0.133     |\n| LoRA + CLIP        | 0.178     | 0.184     | 0.155     | 0.203     | 0.094     |\n| LoRA + CATOD       | **0.245** | **0.203** | **0.217** | **0.244** | **0.178** |\n\nIn this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity. Since the training set contains samples with different angles, it is also easy to produce objects with different angles. We will surely add a visualization of our selected samples and generative results with different poses/angles in our final revision.\n\n**Part 2: Ablating initial size/quality in CATOD (W2, Q2)**\n\nIt is intriguing to ablate CATOD over the initial training set size and quality! In detail, we retest CATOD with different numbers of initial samples (10,20,50) and different quality (high-quality and random sampling), with 100 images sampled at last and 10 images selected per cycle. These experiments are tested with adaptor LoRA. The results are shown as follows:\n\n| Initial Setting         | Axolotl (CLIP) | Axolotl (CMMD) | Emperor Penguin Chick (CLIP) | Emperor Penguin Chick (CMMD) |\n| ----------------------- | -------------- | -------------- | ---------------------------- | ---------------------------- |\n| 10 HQ initial samples   | 73.99          | 0.84           | 74.97                        | 0.71                         |\n| 10 RAND initial samples | 72.89          | 0.93           | 73.95                        | 0.75                         |\n| 20 HQ initial samples   | 75.35          | 0.82           | 74.89                        | 0.69                         |\n| 20 RAND initial samples | 72.27          | 1.10           | 72.38                        | 0.73                         |\n| 50 HQ initial samples   | 75.37          | 0.81           | 74.56                        | 0.63                         |\n| 50 RAND initial samples | 70.39          | 1.34           | 68.85                        | 1.12                         |\n\nFrom these results, we may draw the following conclusions:\n\n1. With HQ initial samples, the initial batch size does not have a significant compact, since a change in this size is not more than 0.3 in the CLIP score and 0.08 in the MMD score. However, this initial size has a significant impact with randomly initialized samples with at most a 5.71 loss in CLIP score and 0.49 in CMMD. We account this phenomenon for that randomly initialized images contain more bad-quality ones that mislead the adaptation.\n2. The quality of initial samples does have an impact on generative results since we can see a consistent performance loss when changing HQ initial samples to randomly initialized samples. With the initial size increase, this impact tends to be even more significant.\n\nThe full results will be added in our revisions.\n\n**Part 3: Ablating the text-to-image encoder (W2)**\n\nThanks for the advice! It is straightforward to extend our experiments to other text-to-image encoders (like SD 1.5 and SDXL). We show some of these results with LoRA in the tables below:\n\n| T2I Model Ablation        | Axolotl (CLIP) | Axolotl  (CMMD) | Emperor Penguin Chick (CLIP) | Emperor Penguin Chick (CMMD) |\n| ------------------------- | -------------- | --------------- | ---------------------------- | ---------------------------- |\n| CATOD + SD 2.0 (original) | 75.35          | 0.85            | 74.89                        | 0.79                         |\n| CATOD + SD 1.5            | 74.79          | 0.88            | 73.01                        | 0.92                         |\n| CATOD + SDXL              | 79.36          | 0.72            | 80.37                        | 0.66                         |\n\nThis table shows that our proposed CATOD is also compatible with different text-to-image encoders with notable performance. We will add these ablations to our final revisions.\n\n**Part 4: How the weight of different scores changes (W3, Q3)**\n\nFollowing Eq. (11) in the main context, the overall aesthetic score $\\gamma_{aes}(A)$, overall concept matching score $\\gamma_{con}(A)$​ and the corresponding weights on concept \"emperor penguin chick\" can be listed as follows:\n\n| Cycle                   | 1     | 2     | 3     | 4     | 5     |\n| ----------------------- | ----- | ----- | ----- | ----- | ----- |\n| $\\gamma_{aes}(A)$       | 6.26  | 8.57  | 9.03  | 9.11  | 9.22  |\n| aesthetic weight        | 0.374 | 0.143 | 0.097 | 0.099 | 0.078 |\n| $\\gamma_{con}(A)$       | 1.86  | 8.59  | 8.54  | 9.35  | 9.67  |\n| concept-matching weight | 0.814 | 0.141 | 0.146 | 0.065 | 0.033 |\n\nDuring the first three cycles, concept-matching takes precedence over aesthetics because the OOD concepts are not yet fully learned by the adapter. As the cycles progress, concept-matching is quickly achieved. In the last two cycles, the adapter stabilizes and focuses more on enhancing aesthetics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N1eF3sMNo4", "reviewer": "Reviewer_uVj3", "summary": "The paper introduces CATOD, a framework designed to enhance the adaptation of text-to-image models for out-of-distribution (OOD) concepts. It addresses the issue of low-quality training data by employing an active learning approach that iteratively improves the training set. The framework utilizes a scoring system comprising aesthetic and concept-matching scores to guide data selection and training. This scoring system dynamically determines the relative weight of the aesthetic score and content matching score to prioritize samples for selecting the training set. The authors demonstrate significant performance improvements using CATOD, achieving notable improvements on standard metrics.", "questions": "1. I wonder if your proposed CATOD also outperforms other methods on previous metrics such as FID and the Inception Score.\n2. In the paper, random sampling and CLIP-score-based sampling are used as the comparison method, and finally 100-200 images are selected for training. Would it be better to fine-tune the model directly using all 1000 images?\n3. In addition to the three methods mentioned in the paper, are there other custom content generation methods that your proposed CATOD framework is still applicable to, such as LyCORIS[1] and ELITE[2]?\n\n[1] YEH, SHIH-YING, Yu-Guan Hsieh, Zhidong Gao, Bernard BW Yang, Giyeong Oh, and Yanmin Gong. \"Navigating text-to-image customization: From lycoris fine-tuning to model evaluation.\" In *The Twelfth International Conference on Learning Representations*. 2023.\n\n[2] Wei, Yuxiang, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. \"Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.\" In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 15943-15953. 2023.", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The research problem of out-of-distribution (OOD) on text-to-image models is essential, and the motivation is quite clear.\n3. The approach is innovative, which combines active learning with a novel scoring system to enhance model adaptation.\n4. The paper has very adequate quantitative experimental results and show clear improvements over baseline methods.", "weaknesses": "1. The paper does not provide enough detail to clarify how the CMMD metric differs from other commonly-used metrics for image generation, such as Inception Score and FID. The authors should further explain this.\n2. Since your proposed CATOD is trained on few samples (100 for single-concept and 200 for multi-concept), the generation model may be biased toward specific samples. Reporting diversity metrics of the generated samples can help clarify this issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank for the good words! We are happy that you enjoyed the paper!\n\n**Part 1: Why CMMD is a good metric (W1, Q1)**\n\nAs reported in many recent works [1,2,3], the most popular image-matching metrics like Inception Score, Precision/Recall, and FID (Frechet Inception Distance) may disagree with human raters, thus ill-suited for evaluating recent text-to-image models. In comparison, CMMD uses CLIP embeddings and Maximum Mean Discrepancy (MMD) that correctly ranks the image sets based on the severity of the distortion. The limitations of FID and previous metrics can be listed as follows:\n\n|                      | FID and other previous metrics                               | MMD distance                                                 |\n| -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ |\n| Inception embeddings | × Weak image embeddings<br />× Normality assumption<br />× Sample inefficient<br />× Biased estimator | × Weak image embeddings<br />√  Distribution-free<br />√  Sample inefficient<br />√  Unbiased estimator |\n| CLIP embeddings      | √ Rich image embeddings<br />× Normality assumption<br />× Sample inefficient<br />× Biased estimator | √  Rich image embeddings<br />√  Distribution-free<br />√  Sample inefficient<br />√  Unbiased estimator |\n\nAlso, we are glad to report our experimental results with FID, Precision, and Recall as follows based on CATOD adapted through LoRA:\n\n| Insect       | FID   | Precision | Recall |\n| ------------ | ----- | --------- | ------ |\n| LoRA + RAND  | 35.02 | 0.55      | 0.83   |\n| LoRA + CLIP  | 28.46 | 0.85      | 0.91   |\n| LoRA + CATOD | 26.74 | 0.92      | 0.94   |\n\n| Penguin      | FID   | Precision | Recall |\n| ------------ | ----- | --------- | ------ |\n| LoRA + RAND  | 11.17 | 0.63      | 0.79   |\n| LoRA + CLIP  | 5.22  | 0.80      | 0.87   |\n| LoRA + CATOD | 4.03  | 0.84      | 0.88   |\n\n[1] Jayasumana, Sadeep, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. \"Rethinking fid: Towards a better evaluation metric for image generation.\" *arXiv preprint arXiv:2401.09603* (2023).\n\n[2] Grimal, Paul, Hervé Le Borgne, Olivier Ferret, and Julien Tourille. \"TIAM-A Metric for Evaluating Alignment in Text-to-Image Generation.\" In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 2890-2899. 2024.\n\n[3] Lee, Tony, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, et al. \"Holistic evaluation of text-to-image models.\" *Advances in Neural Information Processing Systems* 36 (2024).\n\n**Part 2: About the diversity evaluation (W2)**\n\nCATOD maintains the diversity to produce OOD concepts with different angles or poses in generative results. To validate the diversity of our generative results, we further provide a quantitative evaluation with LPIPS~[1,2]. The results are shown in the Table below:\n\n| Comparison Methods | Insect    | Lizard    | Penguin   | Seafish   | Snake     |\n| ------------------ | --------- | --------- | --------- | --------- | --------- |\n| DreamBooth + CLIP  | 0.254     | 0.156     | 0.149     | 0.305     | 0.265     |\n| DreamBooth + CATOD | **0.362** | **0.198** | **0.197** | **0.349** | **0.286** |\n| TI + CLIP          | 0.198     | 0.257     | **0.174** | 0.108     | **0.208** |\n| TI + CATOD         | **0.267** | **0.278** | 0.152     | **0.278** | 0.133     |\n| LoRA + CLIP        | 0.178     | 0.184     | 0.155     | 0.203     | 0.094     |\n| LoRA + CATOD       | **0.245** | **0.203** | **0.217** | **0.244** | **0.178** |\n\nIn this table, we can see that CATOD gives out a diversity improvement up to 0.17 in the LPIPS score compared to CLIP-based sampling, and outperforms CLIP over most categories. From this, we conclude that CATOD also preserves diversity. Since the training set contains samples with different angles, it is also easy to produce objects with different angles. We will surely add a visualization of our selected samples and generative results with different poses/angles in our final revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper aims to improve the adaptation of large-scale text-to-image generative models that allows them to accurately depict the visual
details of out-of-distribution concepts. Inspired by the empirical evidence that the failure cases usually correlate with the low quality of training data, the paper proposes to leverage an active learning paradigm which couples high-quality data accumulation and
adaptor training, enabling a finer-grained enhancement of generative results. A novel weighted scoring system is developed to automatically balance between aesthetics and concept-matching, the two key factors that impact the quality of synthetic results. The overall contribution is supported by both theoretical analysis and empirical results.

Reviewers provided constructive feedback that helped the authors further improve the quality of the paper though the rebuttal process. At the end of the reviewer-author discussion period, all reviewers were in favor of accepting the paper. The authors are encouraged to incorporate the reviewers' feedback into the final version of the paper.
Decision: Accept (poster)