CAS: A Probability-Based Approach for Universal Condition Alignment Score
Authors: Chunsan Hong
ByungHee Cha
Tae-Hyun Oh
Venue: ICLR 2024 spotlight (2024)
Keywords: Generative model
diffusion model
score-based prior
conditional diffusion model
text-to-image alignment score
inversion process
image quality assessment
T2I alignment score
Primary area: generative models

Abstract: Recent conditional diffusion models have shown remarkable advancements and have been widely applied in fascinating real-world applications. However, samples generated by these models often do not strictly comply with user-provided conditions. Due to this, there have been few attempts to evaluate this alignment via pre-trained scoring models to select well-generated samples. Nonetheless, current studies are confined to the text-to-image domain and require large training datasets. This suggests that crafting alignment scores for various conditions will demand considerable resources in the future. In this context, we introduce a universal condition alignment score that leverages the conditional probability measurable through the diffusion process. Our technique operates across all conditions and requires no additional models beyond the diffusion model used for generation, effectively enabling self-rejection. Our experiments validate that our met- ric effectively applies in diverse conditional generations, such as text-to-image, {instruction, image}-to-image, edge-/scribble-to-image, and text-to-audio.

Published as a conference paper at ICLR 2024
CAS: A PROBABILITY-BASED APPROACH FOR UNI- VERSAL CONDITION ALIGNMENT SCORE
Chunsan Hong" KAIST
ByungHee Cha" Seoul National Univ.
Tae-Hyun Oh
Dept. EE and GSAI, POSTECH & Inst. for CREAT, Yonsei Univ.
South Korea hoarer warre@kaist.ac.rr
South Korea paul cha1 025 @snu.
South Korea
taehyun@postech.accm
ABSTRACT
Recent conditional diffusion models have shown remarkable advancements and have been widely applied fascinating real-world applications. However, sam- ples generated by these models often do not strictly comply with user-provided conditions. Due to this, there have been few attempts to evaluate this aligmment via pre-trained scoring models to select well-generated samples. Nonetheless, current studies are confined to the text-to-image domain and require large train- ing datasets. This suggests that crafting alignment scores for various conditions will demand considerable resources in the future. In this context, we introduce a universal condition alignment score that leverages the conditional probabil- ity measurable through the diffusion process. Our technique operates across all conditions and requires no additional models beyond the diffusion model used for generation, effectively enabling self-rejection. Our experiments validate that our metric effectively applies in diverse conditional generations, such as text-to- image, (instruction, image -tomage-----magee edge-/scribble-t--nnee and text-to-audio. Project page: hutps:/uniffied-metti matphiieddmmmerrr.igtubbooooo
Model
Condition
High Rank 13
Luw Rank
Text-4o inage (Stblle
green
hair, wearing sunglasses and dress"
Diffusiun)
Text-to-intage (Fine-tuned Van Gogh)
"use Gugle style, couple sharing umbrella"
Pix2Pix
Samples and Order
"Mike the whatther cloudless
NentrolNet
"Sine wave high pitch"
1
A
Figure I: Results of generated samples ordered by our universal score for various conditions. From left to right, the sequence shows the conditional diffusion model used, the employed condition, two high/low-ranked samples. The last column's generation is audio, so it is visualized using a Mel Spectrogram. Our metric demonstrates automatic and reliable assessment of perceptual alignment between condition and generated results across text-to-image models (Rombach et al. [022 diffu- sion models trained on specific domains such as Van Gogh, InstructPix2Pix (Brooks et al. 2023 ControlNet (Zhang et al. 2023), and AudioLDM (Liu et al. 2023) but not limited to.
'These authors contributed equally to this work.
1 "Corrsppnniigg author.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
The recent surge of large-scale diffusion-based text-to-image models, eg-- (Saharia al.; 2022 Brooks et al. 2023 Zhang et al., 2023] Rombach et al., 2022] Yang et al., 2023 Ramesh et al, 2022). has drawn widespread attention the overall conditional diffusion models. The conditional diffusion models take arbitrary conditional inputs provided by the users and generate impressive samples conforming the given conditions. While text-to-image is the most popular example, the modalities and types of conditions are not limited, e.g., image-to-in to-image (Brooks et al. (023), pose- /segmentation-to- -on-mmaiiioo--- (Zhang et al.] 2023), text-to-audio (Liu et al. 2023), etc. These develop- ments exhibit impressive compositional generalization abilities and complying behaviors according to unbounded conditions. Despite those impressive quality, diffusion models generate undesirable samples that are not perfectly aligned with or even further neglect the given conditions. In this casc, the common practice is to generate many samples for a given condition and manually pick preferred images among the diverse samples. This demands human's 's effort, cost and time.
To mitigate these costs and automate the processes, in the specific text-to-image (T2I) application, numerous learning-baan -aased alignment scores (Hessel al., 2021 Wu et al. 2023 Xu et al. 2023 Kirstain et al. 2023) have been proposed, where they train a separate neural network model to predict perceptual alignment score between text and an image. Likewise, there are few attempts
to define individual scores for each problem depending on different forms of the condition and the
output, e.g.. Gal et al. (2022) present the directional CLIP similarity for image editing.
The previous development methods are all anchored for each specific format pair of condition, in- put, and output. These require to design alignment scores individually for various conditions, such as text, segmentation, pose. and others. Establishing a new alignment score for each of the count- less types of condition pairs requires an immense amount of effort. In particular, learning-based approaches, e.g., (Hessel et al. [2021; Wu et al., 2023, Xu et al. 2023 Kirstain et al. 2023), strongly rely on extensive and well-curated data for training neural models, which demands significant ex- penditures in terms of both cost and time for data collection and model training. Furthermore, there are practical scenarios, where collecting data and training a new score model are even more chal- lenging; for example, specifically to the T2I field, conditional diffusion models are often fine-tuned for various domains for sperifie needs, such as animation, pixel art, or even the distinctive style of a particular artist. However, our experimental findings suggest that existing metrics, relying on pre-trained models, markedly underperform in these scenarios.
In this regard, we propose Condition Alignment Score (CAS), the universal score evaluate align- ment with the given condition and generated samples, regardless of the type of conditions and with- out any separate model training and data. Our key idea is to use the conditional likelihood p(r]c) which can be obtained from the diffusion model as CAS. We utilize the same diffusion model which used in generating samples without modification, i.e. self-rejection, thereby addressing all the previously mentioned shortcomings. By leveraging the diffusion model itself, no pre-trained scor- ing model is needed, and our technique possesses the capability to reject out-o--distribution images from the domain in which the diffusion was trained. Since the conditional probability itself is used as metric, there is no need to design separate scores for each condition individually.
We leverage the exact probability computation method by Song et al.] (202Ib), yet the condi- tional probability is not measurable during the sampling process where Classifier Free Guidance (CFG) (Ho & Salimans, 2022) is employed. Therefore we utilize the inversion process. However, we observe that the common practice of the inversion method (Song et al.] 2021a). called Denoising Diffusion Implicit Model (DDIM) inversion, does not provide accurate alignment of the reverse pro- cess due to the utilization of approximations. We additionally present a recursive inversion method which improves the performance of our method. Our main contributions summarized as follows:
We present a universal alignment score applicable to all possible conditions.
Our self-rejection technique leverages the diffusion model that is used for generating images, thereby eliminating the dependency on well-curated datasets and separately pre-traines score mod- els that the previous studies grappled with.
We developed method for improved DDIM inversion, boosting the performance of our technique.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORK
Since our scope is related to the assessment of the alignment between given conditions and gen- erated samples, we first brief the evaluation conventions used in the most popular application of diffusion, text-to-image (T2I). Among T2I diffusion models, Imagen (Saharia et al. 2022), Sta- ble Diffusion [Rombach et al. 2022), and DALL-E2 (Ramesh et al., 2022) stand out as the most prominent. In these studies, the primary metries proposed for T2I alignment largely rely on human evaluation and the CLIP Score (Hessel et al.) D. CLIP Score measures the cosine similarity be- tween the text representation and the generated image representation obtained from the pre- pre-trained CLIP model (Radford et al. (02T). The most recent studies (Wu et al. 2023] Kirstain et al.] 2023] Xu al., 2023) propose learning-
metries in which training data includes prompts, multiple images generated from each prompt, and their human preference annotations. All the aforemen- tioned scores rely on learning models pre-trained on elaborated or vast datasets. Beyond the T2I diffusion models, various kinds of conditional diffusion models such as ControlNet (Zhang et al., 2023), InstructPix2Pix (Brooks et al. 2023), and AudioLDM (Liu et al.) 2023) have been proposed with different combinations of modalities for conditions and generated targets. Considering vast application scenarios, it would be impractical to define and develop respective scores for each com- bination case; thus, it has been barely studied for general use cases.
Our work focuses on measuring the conditional probability of generated samples, and for this, we employ the method proposed by [Song et al. (2021b). They present method to measure the log- likelihood of images through a pre-trained diffusion model, which has been adapted in various stud- ies (Zimmermann et al.] 2021 Feng et al., 2023). Zimmermann et al. (202T) utilize this method to measure the class-conditioned likelihood of images on CIFAR-10 10 for classification,, achieving promising results. However, find that calculating conditional probability in this manner does not work as desired in complex conditions, such as natural language prompts. Our approach diverges in that we additionally exploit unconditional probability for debiasinn, thereby expanding the effective- ness to numerous conditions, Feng et al. (2023) utilize their method to solve the inverse problem and empirically exhibit that the image prior is measured close to the real- worldworld prior. However, while they aim to solve problems using the prior, our approach differs in that we strive to solve problems leveraging the likelihood.
Research measuring the likelihood probability is not only confined to diffusion models, but has been explored in other generative models, such as flow-based generative models. Flow-hased generative models (Dinh et al.] 20155; Rezende & Mohamed, 2015 Kingma & Dhariwal 2018) come with a method to calculate the likelihood of a sample and utilize it as an objective function in model training. Nevertheless, this approach bears critical drawback: it imposes substantial constraints on the architecture of models to satisfy the required flow properties. This limitation hampered the development of flow-based generative models for practical applications, which is why utilize diffusion models. Nonetheless, this remains a notable area, as the method proposed by Song et al. (021b) is inspired by the formula that flow-based generative models aim to solve.
3 PRELIMINARY
DDIM and DDIM inversion. Denoising Diffusion Probabilistic Model (DDPM) (Ho et al. 2020) serves as generative models that aim to approximate real data distribution g(x) by a model pe (x). For each Xo ~ (x).. DDPM constructs a discrete Markov chain {Xo, XN} that satisfies [[xx(x(11 N(x;; /1 3ex,-1, B,I). This process is termed the forward step, where the se- quence {5. [ is a sequence of positive noise scales, i..., 0 < 31, 32..... 3x < 1. Assuming that N considerably large, DDPM presumes that (x) & NO; DDPM starts by sampling x,v from a Gaussian distribution and then undergoes stochastic reverse process to generate the sample Xg. e.g., image.
On the contrary, the Denoising Diffusion Implicit Model (DDIM) adapts the formula to facilitate a deterministic reverse process, enabling inversion. Given the conditions of the forward step, the following equation is satisfied: x, = VazXa + v1 - ayc, e ~ (0; I), where a; = II (1 - 3²). By training the diffusion model 0 to predict noise that is deviated to the original image x, with given x,, f.e. training with the objective function E[] (X,,, - el ell²], we can perform the following sampling to generate image:
3

---

Published as a conference paper at ICLR 2024
X1 vI (1 + v 0t-1 - ox -1-ox-1-ofeo(x;)) oeet.
X va+
Va
Setting o₁ for all results deterministic DDIM sampling process. When rearranging this equation for x,, it becomes the ideal DDIM inversion, which recovers the latent from the image:
X = at- 1 + 1 (x:: t),
where 0:-1 to use 2i itself to recover Tt. Therefore, is convention to approximate it by:
Vaa vare-
b = (V1 04
Gr 0n), and or = 0. However, is contradictory
X - + b1-16((x
(3)
so that X x &t This is referred to the DDIM inversion. The effectiveness of this approximation hinges on the linearity assumption to the latents, which is proved by Miyake et al. (2023).
Probability Flow ODE and Exact Probability Computation. Song et al. (0211) present proba- bility flow ordinary differential equation (ODE). Contrary to methods like DDIM and DDPM, where I treated discrete variable, in probability flow ODE, is considered as continuous variable, where latent is denoted as x()). We change the domain of from [1, N] to [o, 1] for probability flow ODE. Then, probability flow ODE given as follows: dx (x(((t), t)dt. By applying the
instantaneous change of variable formula provided by Chen et al. (2018) to this equation, one can compute the probability value of the synthesized sample x(0):
log = log P1 (x(1) + /'' Vx fo(x(t),
(4)
where Pı (x) Re (0; Since computation of Vx (x) is intractable, Song et al. 2202 lb) ap-
proximately compute this value by Skilling-Hutchinn trace estimator (Skilling 1989 Hutchinson 1989 : Vx fo [x, t) RJ Ep(z)[z (xf t)z) z is a random variable which satisfies Ep(x)
and Covp(2) [z] I, and Vx is Jacobian matrix of fo where z Vxfo((x, can be computed by reverse mode automatic differentiation.
Classifier Free Guidance (CFG). When training a conditional diffusion model, learning solely
with [x], c,
e { does not yield satisfactory results. Thus, CFG (Ho & Salimans 2022 is commonly employed. In this method, both [x 0, - <ll²) and E]]leo(x,, c, () <
are simultaneously trained within single diffusion network, and an image is sampled through the following equation: éx(x, c, (x 0, f) + w (Ee(x, c) c0 0, t)). We will denote the latent variable derived through CFG as {
METHOD
4.1
METRIC DEFINITION
Our goal is estimate the true conditional likelihood logp(x/c) of the generated sample x given condition c as fundamental metric for CAS. In this section, delve into the rationale behind the decision of our score formu- lation defined by probability. The methodology for cal- culating probability will be discussed further in Sec. 4.2
Comparison between Varions Probs.
Mean Clip Score
log pa(atc]
3.0
lon pu(a).
We approximate the conditional likelihood Po (x|c) p(x(c), by the diffusion model as competent metric for measuring whether generated samples satisfy the given condition. However, upon ranking with this metric, we discovered that it did not exhibit meaningful tendencies. This lack of meaningful pattern would be primarily due to the diffusion model's 's inability to mimic the true likeli- hood accurately.
log
2.9
lagpe(
logps(ar
100
Cherry pick N% via each method 2: Average CLIP Score (Hessel et al. 2021 of top N% images cherry-
Figure
picked by each metric.
4

---

Published as a conference paper at ICLR 2024
Step Sampling Process (CFG)
Step 2: Inversion Process
Step Compute Probubility
Ex-1
X1
pxalec)
N.c)
Ep (3p 1,c)
Ep (x 0,c)
Ep (Ag V. 1,
Initial Noise Xn
Generated Image Xg
Prompt "Woman with green hair, wearing sunglasses"
p(x))
(Xo, 0, Ø)
Ep (50 1,0)
Figure 3: The figure illustrates method and which latents of the diffusion process are used to compute CAS, PogPo(Xo c) - log po The values highlighted in yellow in Step 2 are needed. Therefore, measure CAS, the image first undergoes inversion process (Step 2), and then the values obtained during this process are used to calculate pe (xo) and pe (xolc) (Step 3).
In our analysis, we found a significant bias stemming from px(x) when calculating pe (x(c). We conducted preliminary experiment using the Stable Diffusion v1.5 model Rombach al. (2022) in Fig. 2 We synthesized 100 images from the prompt "Woman with green hair, wearing langlasses
and dresses." We then compared the average CLIP Score (Hessel et al. 2021) of top N% images therry-picked by each of log pe (x), log pe (x), log pe (x c) and log Pe x x log (x(x.. Fig. 2 illustrates that images boasting a higher log po (x)) tend to possess markedly lower CLIP Score, and that the tendency of log (x c) and that of log Pa (x) are almost the same. Since a lower CLIP Score indicates that the image not aligned with the prompt, this advocates the hypothesis that log (x(cc) c) does not mimic the true likelihood, p(x|c). On the other hand, we observed log (x) shows better performance than log po (x). Therefore, we venture the following hypotheses: 1) the conditions for generating images are often unseen in the training data of the diffusion model; hence resulting in a low measurement of p(x) and 2) this effect is so pronounced that it biases p(x(c). We speculate that the task of forming the likelihood distribution of an "unscen condition" demands strong extrapolation assumptions, which in reality may not hold true. [Sewwag et al. (2022) demon- strate the difficulty in probability estimation in diffusion models when faced with unseen images, Kirice/Krichenkoo et al. (2020) reveal similar challenges within flow-based generative models. Based on these findings, we hypothesize that extrapolation for probability computation on unseen condi- tions may pose challenges in our scenario.
To address this inherent bias, it is imperative correct the root cause, which is the skewed log po (x). To this end, we introduce a correction factor into our ranking metric, S = log pe (x(c) A log pe (x). In our experiments, we set A 1. With this setting, it is equivalent picking a sample based on log ()(x) where p(c) is constant. As shown in Fig. 2 images with higher values of our metric tend to align well with conditional text description.
4.2 DDIM IN ODE FORM & PROBABILITY COMPUTATION
we define CAS and present its computation method utilizing equa- tion|4 We denote the latent generated from the unconditional inversion process at time step t as x and the latent generated from the conditional inversion process at time step as xf.
Based on the idea of Sec. 4.1
Both (x) and (x|c) cannot be measured in the reverse process since CFG breaks the stochastic modeling of the diffusion model. Instead, the probabilities can be obtained during the inversion process which is the analogous process to recovering latents. In Fig. 3 we illustrate the process of how we obtain the latents necessary for probability computation from conditional diffusion models. When N is large, by transferring the time domain from [1, N] to [0, 1]. we can convert x and x; into continuous variables with respect to t as x" (t/N) and (t/N). Accordingly, CAS is defined:
Definition 1 (Condition Alignment Score (CAS)). Given diffusion model, O, the latent at time from an unconditional DDIM inversion process initiated from an image x(0). x² and the corre- sponding latent at the time from a conditional DDIM inversion process also beginning from x(0), x (I), the CAS is defined as:
P1 ( (x)() (1) -f d'(t) Vx (x.co(x(t).c Vx co(x" (t),0)) dt. (5)
CAS(x, 0) -log
2x(() a(t)
5

---

Published as a conference paper at ICLR 2024
We can compute equation using numerical solver. To derive the above metric, start by repre- senting the DDIM in the form of an ordinary differential equation (ODE), (the detailed derivation is provided in Supplementary Material)
dx
x(t)
(x(x))),t 2a(1) v -a(()
at())
(6)
di
2a(1)
Applying equation 4 to equation 6 we can compute the log probability in DDIM inversion process:
va(l D- !. a'(t) Vx ce.co(x" (t), (t) dt,
log po (x (0) log (x (1)) + log
Va(1)
I-(()
where is the number of dimensions x. For given condition c, the conditional log probability expressed as:
log (x = logpi(x + log va(0) (a
/ a'(1) Vx (x(x dl.
D
(8)
20(t)/1-a(()
By subtracting equation from equation 8 we derive the formula for CAS in Definition ! which
offers principled framework for evaluating the alignment between sample and given condi- tion through calculating associated probabilities. Note that applying CAS not limited to image domains, but can be applied to arbitrary modalities of input and condition.
4.3 IMPROVING DDIM INVERSION PROCESS VIA DDIM RECURSIVE INVERSION
To obtain accurate measure of probability, the inver- sion process must perfectly align with the reverse process. However, the current DDIM inversion does not, as an er- ror occurs when approximating
Error of Each Inversion
X, (Mokady et al.)
MSE(6" [x)
2023 Miyake al. 2023). Although several studies
DDIM (n 1)
1.0
have been conducted to address this issue in image edit-
Recur. (a=2)
ing, Mokady et al. 2 Miyake et al. (2023) present the compensation methods exclusively for image editing rather than accurately calculating the inversion. On the other hand, WWlllace et al. (2023) defines precise method for inversion using two coupled variables. However, it is unsuitable for probability computation, since it is neces- sary to solve the coupled variable differential equation. In contrast, we precisely invert the DDIM without bells and whistles. Under the simple assumption that x, is closer to x, than X we can establish the following equation:
0.5
Recur (u=5)
0.0
200
400
600
800
1000
step #
Figure 4: Mean Squared Error (MSE) of latent during the reverse process and inversion process of the original DDIM inversion and method (recur- sive inversion). We sample XN from a Gaussian distribution and then gener- ate [= through reverse process. Subsequently, starting from Xo, we im- (9) plement the inversion processes to pro- duce {&] 1.V Ji=1, where setting n becomes original DDIM inversion and n > becomes our recursive inversion.
Definition (DDIM Recursive Inversion). We define the i-th step recursive inversion as:
x a by 1
for 2,3,.. ... and x7 St. We denote the approxima- lion R X2 the nith order recursive inversion.
The difference is apparent by comparing with the original DDIM inversion X, - + (1-160(x,-11) t). To verify the validity of Definition 2 we empirically test on the CelebA dataset, as shown in Fig. 4 This preliminary test exhibits that, through mere utilization of our proposed equation, the error between the inversion process and the reverse process can be diminished less than 10% of the original DDIM inversion, i.e., accurate. In addition, one might question whether simply increasing the number of the inversion steps could be beneficial. From our experiments, the recursive inversion not only saves much time compared to just increasing the steps but also hances the deerrr- performance of CAS more. We provide the evidence for these claims in Supplementary Material.

---

Published as a conference paper at ICLR 2024
0.1
0.01
0.001
0.0001
0.00001
NRMSE
0.164
0.012
0.002
0.021
0.212
Table 1: Normalized Root Mean Square Error (NRMSE) between log probability computed ing Song et al. 2021b) and log probability computed using our method. NRMSE is defined as
N NRMSE( VINN E, ((vs 1,) where is ground truth and y is approximation.
1 Van Gogh Image
4 Non Van Gogh Images
Figure 5: Example of generated images from Van Gogh Diffusion. The prompt used for generating images is "Van Gogh style, romantic gondola ride under Venetian bridges". However, except for the leftmost image, negative weighting to "Van Gogh style" was done for the rest of the images, leading to noticeable deviation from the intended artistic style.
4.4 FASTER COMPUTATION
In this section, we propose an efficient and faster method to approximate
trace estimator (Skilling 1989 Hutchinson 1989) Ep(z)[z (x, t)z]. Song et al. (2021b) bypass Vx60 (x, () us-
Jacobian computation from ing backpropagation. However, backpropagation requires expensive computations. To avoid the resource-intensive backpropagation process, we have approximately computed z Vxee(x, t)z as:
Hutchinson estimator by directly calculating z
co(x + oz.t) cx( 1 -z (Eo(x oz,t) (x,,
z Vxee(x,t)z=> TT
(10)
where is sufficiently small number. By leveraging the above expressions, can make the approximation Expz)) Vx (x, t)z] feasible even without resorting to backpropagation. Notably, with only two forward operations, the process becomes significantly faster, reducing the total time to 16% with negligible error. Furthermore, GPU memory used for backpropagation is also saved. In this work, we set and fix 0.001, which shows the best approximation of log probability in our empirical test as shown in Table 1
5 EXPERIMENTS
In our experimental section, we validate the efficacy of our approach through tests on five distinct models: a T21 diffusion model fine-tuned in the Van Gogh domain (Mackay) 2023). Stable Diffusion v1.5 (Rombach et al.) 2022), InstructPix2Pix (Brooks et al. 2023). ControlNet (Zhang al. 2023), and AudioLDM (Liu et al.] 2023). Through experiments, we validate that CAS consistently delivers high performance in various domains without necessitating additional model training.
5.1 T2I DIFFUSION MODEL
Baselines. To validate ability of CAS as T2I alignment score, we selected several baselines: CLIP
Score (Hessel et al. 2021). Human Preference Score (HPS) (Wu et al. 2023), Image Reward Xxu et 2023), and Pick Score (Kirstain et al. 2023). CLIP Score leverages CLIP (Radford al. 2021 model which is trained with real image and text annotation pairs. In contrast, HPS, Image Reward, and Pick Score utilize models that are trained based on human-ranked annotations of multiple images generated from text prompts. Details are shown in Table Our method utilizes 2nd order recursive inversion with total step of 10. The comparison between our method and the established baselines

---

Published as a conference paper at ICLR 2024
Image Reward
CLIP Score
HPS
Pick Score
Ours
Train data (Image) Train data (Text)
400M
4
999
~584K
None
400M
~9K
~25K
~38K
None
Acc. Van Gogh dataset
0.284
0.247
0.229
0.338
0.470 = 0.013
Acc. on Pick Score dataset
0,580
0.621
0.697
0.721
0.622 + 0.004
Table 2: Amount of training data for each metric, and accuracy on our conducted Van Gogh dataset and Pick Score dataset. For our method, accuracy is measured five times, and we indicate the 95% confidence interval for these measurements.
Model
InstructPix2Pix
Canny Edge ControlNet
Scribble ControlNet
AudioLDM
Accuracy (%)
59.2
64.3
61.0
58.3
Table 3: Human preference evaluation on each model. The accuracy listed in the table represents the average preference accuracy across five participants.
hinges two pivotal criteria: 1) The capacity to measure whether the image has style that the diffusion model is fine-tuned on, and 2) The accuracy in evaluating T2I Alignment. Each of them is measured under the experiment done in conducted Van Gogh dataset and Pick Score dataset.
Experiment Setting in Van Gugh Dataset. Evaluating the style fidelity in domain-specific diffu- sion models is important. Typically, this style is specified within the prompt. Therefore, the T2I alignment score should also be able to measure whether the image adheres to the specified style. To exemplify this, we conduct experiments Gogh, the well-known artist. For evaluation, we utilize 275 prompts, randomly generated by GPT-4, encompassing landscapes, portraits, and still-life paintings. For each prompt, we generate five dis- tinct images using the Van Gogh diffusion model, denoted as "Van Gogh style, (prompt)". Images are divided into two groups: 1) one image generated directly from the prompts themselves, and 2) four images generated by applying negative weighting (implemented by the diffuser (von Platen al. 2022)) to the "Van Gogh style" portion of the prompt. We designated the ground truth for the first group as Van Gogh and for the second group as von-Van Gogh. Additionally, we conducted a human verification step. Examples of this can be found in Fig. S where all samples are clearly distinguished. We compute the proportion of numbers each metric successfully identifies one real Van Gogh style image per each prompt. The prompts used for evaluating T2I Alignment in each metric also framed "Van Gogh style, {prompt)".
diffusion model specifically trained Vincent Van
Experiment Setting in Pick Score dataset. The Pick Score dataset consists of a total of 500 prompts, two image pairs generated from each prompt, and annotations based on human preferences. Among these, data entries marked as "indistinguishable" are excluded, and we analyze the accuracy of each method in the remaining 432 image pairs. We utilize Stable Diffusion v1.5 for our score.
Results. As shown in Table 2 our approach exhibits an accuracy that is approximately 13.2% higher than the second-best performing method, Pick Score in Van Gogh dataset. This result underscores the inherent advantage of our self-rejection technique, an exceptional capability to measure whether images are from the intended domain. The margin of superiority already notable; however, given that Van Gogh is a globally renowned artist, it is plausible to infer that our method's performance would be even more pronounced minor domains. We also observed that our method outperforms CLIP Score and is comparable with Image Reward in Pick Score dataset. While our accuracy might be lower than that of HPS or Pick Score, the fact that CAS outperforms the CLIP Score and is com- parable with Image Reward without using any pre-trained scoring models is remarkable. Although it might not represent the state of the art in the T2I domain, the significance lies in the potential for saving time and resources in the majority of other conditional domains where scores have not yet been researched. The 2nd order recursive inversion achieved an accuracy about 2.9% higher than the original DDIM in Pick Score dataset, 1.1% higher than the original DDIM in the Van Gogh dataset.
8

---

Published as a conference paper at ICLR 2024
5.2 ADDITIONAL MODALITIES
To corroborate the universal applicability of our formulated metric, we conduct human preference evaluations across multiple modalities including InstructPix2pix (finstruction, image)-to-imagee ControlNet (fimage, text)-to-image), and AudioLDM (text-to-audio).
Human Preference Evaluation. Given the challenge of finding quantitative metrics for evaluation, we resorted to human preference evaluation to validate our metric's efficiency. First, we conducted datasets composed solely of specific conditions, e.g. the dataset for ControlNet evaluation includes 100 canny edge images. We then generate 100 samples for each condition in the dataset. Subse- quently, participants are given a condition and pair of two images generated by the corresponding condition: 1) A randomly selected sample, and 2) sample with top-1 CAS from the 180 samples. They are asked to select sample that aligns with the condition more. Subsequently, we recorded the ratio of instances in which participants opted for the top-ranked generated sample.
Experimental Setup in InstruetPix2Pix. We selected 180 random test cases from the Instruct- Pix2Pix dataset, each consisting of an original image and prompt for image editing. Since Instruct- Pix2Pix is a multi-condiitonal diffusion model, we add a slight twist to our metric: logpe (ogpo( Cp) log Po (x(c,, where c, is the original image and cp is the editing prompt.
Experimental Setup in ControlNet Model. We evaluated two variants of image conditions: a scribble image for Scribble ControlNet and canny edge image for Canny Edge ControlNet. To carry out the evaluation, we assembled 180 original images from InstructPix2pix. For each original image, we extracted both seribble and canny edge images. We used extracted image conditions test dataset (edited prompt was employed together to generate samples).
Experimental Setup in AudioLDM Model. To investigate our metric's efficacy in text-to-audio modalities, 180 prompts are generated by GPT-4 and used for human preference evaluation.
Results. The results in Table 3 demonstrate around over 60% preference for our metric across modalities. Particularly for conditions like canny edge and scribble, despite their inherent ambiguity in alignment measurement, we observe notable congruence with human perception. Additionally, the results from AudioL.M demonstrate that our approach functions effectively, even when the diffusion model output is not an image. Through these experiments, we verify that our method successfully measures alignment across any input-output pairs of the diffusion model.
Comparison with Directional CLIP Similarity in Instruct- Pix2Pix In the case of 2ntructPix2pix, we additionally compare the relation between CAS and directional CLIP Similarity. The direc-
Aus Diretimal CLJ simburit
0.20
tional CLIP similarity, introduced by Gal et al. (2022), measures the cosine similarity between the embedding diffcrentials of original and edited images and texts in the InstructPix2Pix model. This met- ric is also employed for the evaluation of InstructPix2ix in [rooks] al. (2023). To compare our metric with directional CLIP simi- larity, we assess for average directional CLIP similarity per rank. The outcomes in Fig. 6
0.19
1/
0.18
25
50
75
100
Rank
inndicate an inverse proportionality between our metric and directional CLIP similarity. with higher rank in our metric corresponding to greater directional CLIP similarity.
Figure 6: Average direc-
tional CLIP similarity based
on CAS rank in the test data.
CONCLUSION
We have successfully introduced CAS, that leverages conditional probability calculate alignment between generated samples and its corresponding conditions. We find that the conditional proba- bility Pe (x(c) computed from the diffusion model is biased toward probability pe (x). By utilizing this knowledge, we achieve success in measuring condition alignment directly from the diffusion model. Furthermore, we present the recursive inversion which reduces the error of DDIM inversion to within 10%. Additionally, by presenting an approach that approximates backpropagation compu- tation with minimal error, we have also managed to reduce computation time to 16%. Through these methodologies, our technique demonstrates high performance across various conditions without the need for pre-trained scoring models. Our research is readily applicable across all domains utilizing the conditional diffusion model, substantially reducing the resources dedicated to cherry-picking.
9

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENT
T.-H. Oh was partially supported by Institute of Information & communications Technology Plan- ning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 200.2021-0-02068, Artifi- cial Intelligence Innovation Hub; No.RS-2023-00225630 Development of Artificial Intelligence Development of Artificial Intelligence Technology for Self-Improving Competency-AWw Aware Learning Capabilities) and by the National Re- search Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. NRF. 202IRICICIIO DD2IRICIC100699),
for Text-based 3D Movie Generation; No.2022-0-O06
ETHICS STATEMENT
Generative models for synthesizing images carry with them several ethical concerns, and these con- cerns are shared by (or perhaps exacerbated in) any generative models such as ours. Generative models, in the hands of bad actors, could be abused to generate disinformation. Generative models such as ours may have the potential to displace creative workers via automation. That said, these tools may also enable growth and improve accessibility for the creative industry.
REPRODUCIBILIIT STATEMENT
We will release the code and examples used in this work.
REFERENCES
Dreamlike Art.
Theamlike-photoreall- 2023. URL httpa::/hucginggeee Accessed on 2023-11-13.
dreaml ike-art wttpa///rrraaa.ieiaeoooeeeaaaaaa
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image
editing instructions. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: non-linear independent components esti-
mation. In International Conference on Learning Representations (ICLR), 2015.
Berthy T Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William T Freeman. Score-based diffusion models as principled priors for inverse imaging. In IEEE International Conference on Computer Vision (ICCV), 2023.
Rinon Gal, Or Patashnik, Haggai Maron, Amit Bermano, Gal Chechik, and Daniel Cohen- Stylegan- nada: Clip-guided domain adaptation of image generators. ACM Transactions on Graphics, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metrie for image captioning. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2021.
Jonathan Ho and Tim Salimans. arXiv:2207. 220.. 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Michael F Hutchinson. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Communications in Statistics Statistics-Simulationn and Computation, 1989.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl convolutions, In Advances in Neural Information Processing Systems (NeurIPS), 2018
10

---

Published as a conference paper at ICLR 2024
Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect out-of-distriuution data. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for texi-to-image generation. arXiv preprint arXiv:2305.01569, 2023.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301. 2023.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations (ICLR), 2022.
Dallin Mackay.
Van-gogh-diifusion, 2023. URL. htttps//huuggiggfant co/ on. Accessed 2023-09-27.
da 1 i iminacaaay Maaeeckaayyaa-Goo-iiiii
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka. Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models. arXiv preprint arXiv:2305. 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In IEEE Conference Computer Vision and Puttern Recognition (CVPR), 2023.
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Prompthero.
Openjourney, 2023.
URL httpa://haeginngfaecccc///ooreoe
openjourney Accessed 2023-11-13.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows In Interna- tional Conference on Machine Learning (ICML), 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic tixt-to-image diffusion models with deep language understanding. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2022.
Vikash Sehwag, Caner Hazirbas, Albert Gordo, Firat Ozgenel, and Cristian Canton. Generating high fidelity data from low-density regions using diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
SG1612222023. Realistic vision v1.4. 2023. URL httposs//ouggingfaaececo///11222 Reali iillisst__iiiio____
Accessed on 2023-11-13.
John Skilling. The eigenvalues of mega-dimennional matrices. Maximum Entropy and Bayesian Methods, 1989.
John Slegers. Epic-diffusion, 2023. http:sr/mogggiaaaa....oooossssst poic-diffuse on Accessed 2023- 11-3.

---

Published as a conference paper at ICLR 2024
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations (ICLR), 2021a.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations (ICLR), 2021b.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-ar diffusion models. https:
hrrpthuww..o/mnggggggacc////ee/e httpg/ffaa.ffffffff/////aa 2022.
Bram Wallace, Akash Gokul, and Nikhil Naik. Edict: Exact diffusion inversion via coupled trans- formations. In IEEE Conference on Computer Vision and Patern Recognition (CVPR), 2023.
Xiaoshi Wu, Kegiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. IEEE International Conference on Computer Vision (ICCV), 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.05977, 2023.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
Roland S Zimmermann, Lukas Schott, Yang Song, Benjamin A Dunn, and David A Klindt. Score- based generative classifiers. Advances Neural Information Processing Systems Workshops (NeurIPSW), 2021.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
log Po (z)
log (e (2)
log po (x)
CAS((z c1. c1. 0)
CAS(xx, C22 0)
21
61059.39
61046.07
61011.86
47.53
34.21
222
59566.22
59612.24
59546.12 12
20.10
66.12
Table 4: The average log po (azle), log (a)), and (a, c, 0) of each 100 images œ and æ2 generated from C₁ "a woman with black hair" and ="a woman with rainbow hair".
Employed 0 in CAS(x,
Image Source
DP2.0
OJ
SD1.5
Real
DP2.0 OJ
189.90
140.88
32.39
10.47
121.80
171.81
51.04
27.85
SD1.5
72.50
54.13
163.24
36.79
Table 5: Average CAS values for images from multiple sources. We measured average CAS for 100 real images from the Coco dataset and 100 images each generated using Dreamlike Pho- toreal 2.0 (DP2.0) (Art) 2023), OpenJourney (OJ) (Prompthero (2023), and Stable Diffusion (SD1.5) (Rombach et al., 2022), based on 100 captions from the COCO dataset. For the CAS mea- surement, the same diffusion models, DP2.0, OJ, and SD1.5 are employed.
A FURTHER ANALYSIS OF CAS
A.1 ADDITIONAL INVESTIGATION INTO PROBABILITY SPACE OF THE DIFFUSION MODELS
In Sec. we argued that 1) since the user-specified condition is usually not observed in the training procedure of diffusion, an image properly generated from a condition actually of distribution in the training distribution. resulting in a lower measured pe (x), 2) this effect is so strong that it biases pe (x c), and 3) by debiasing the conditional probability with the probability, log Po (x), it becomes possible to measure alignment between image
i... CAS = log Po (x|c) and conditions. In this section, we provide toy experiment to support these assertions. We se- lected two prompts, one representing an unlikely scenario in the real world and the other a more plausible scenario. The first prompt, C₁ is "a woman with black hair," and the second, C22 is "a woman with rainbow hair." Then, we generated 100 images from each prompt and measured their average log Po (x/c), log (x)). and CAS((x, c, 0). As shown in Table the results satisfy all aspects of hypothesis: 1) E[log (o >> E[log pe (x²)], indicating that less observed condi- tions result in a lower probability, 2) the similarity between E[log (x)] and E(log Po (x (x|c)] sug- gests that log popoooox (x|c) is biased by log po (x)), and 3) ESCAS((), C12 0)] TECAS (x2, C1 f)] and E[C' ASC', C22 0)] > B[CAS((x1, C22 0)) demonstrate that CAS effectively measures the alignment between images and conditions.
A.2 THE ABILITY OF CAS TO DETECT OUT-OF-DISTRIBUTTI SAMPLES
In this section, we further investigate the capability of CAS to detect out-of-distribution samples and apply this to fake detection and image generation source detection. In Table Sablee[]on our technique significantly outperforms other methods in the Van Gogh dataset. This indicates the superior ability of our approach to identify samples that are out-of-distribution from the training distribution of the diffusion network. To further analyze this, we design the following experiment. We generated 200 images from each of the three different diffusion models, Dreamlike Photoreal 2.0 (DP2.0) (Ari, 2023), Open Journey (OJ) (Prompthero) 2023). and Stable Diffusion 1.5 (SD1.5) (Rombach et al, 2022 We have also prepared 200 real images. Then, we measured the average CAS of each group of images according to the diffusion models for image generation, and CAS is computed with respective diffusion models that are used for generating each group of images. As shown in Table 5 CAS is significantly higher for images generated by the same diffusion model that CAS employs. Therefore, in the Van Gogh dataset, the scores biased towards the Van Gogh images,
13

---

Published as a conference paper at ICLR 2024
Detection Type
Image Source
Category
Real
D22.0RV..AVPPII/DD.II
SD1.5/OJ
# of
100
100 per each model
None
Source Detection
Train Samples
# of Test Samples
200
200 per each model
None
100
100 per each model
None
Fake Detection
Train Samples
# of Test Samples
200
None
200 per each model
Composition of train and test data used for source detection and fake detection. For dataset construction, we randomly sampled 300 (caption, image) pairs from the COCO valida- tion dataset. For each caption, we generated images using six diffusion models: Dreamlike Pho- toreal 2.0 (DP2.0) (Arl 2023). Realistic Vision 1.4 (RV1.4) (SG1612222023 2023), EPIC diffusion (EPIC) [Slegers 2023), Open Journey (OJ) (Prompthero 2023), Stable Diffusion 1.5 (SD1.5) (Rom-
Table 6:
bach al. 2022), and Stable Diffusion 2.1 (SD2.1) (Rombach al. 2022), and then divided them into train and sets. sets.
Method
Accuracy
Average Precision
CLIP-VIT (LC)
0.87
0.91 0.92
Ours
0.96
Table 7: Result of fake image detection. CLIP-VIT(LC) OOjha et al. 2023 is the current state-of- the-art in fake detection.
Source
Real
DP2.0
RV1.4
EPIC
SD2.
Total
Accuracy 0.87
0.86
0.90
0.92
0.95
0.90
Table 8: Result of image generation source detection.
leading to higher CAS accuracy. Furthermore, the characteristic of CAS producing high values for in-distribution samples and low values for real images shown in Table 5 can be straightforwardly applied to fake image detection and image generation source detection. Through small-scale periment, we show the potential of our method to surpass the current state-of-the-art (Ojha et al. 2023) in fake detection.
For fake detection and image generation source classification, we built a small custom dataset by randomly sampling (image. caption) pairs from the COCO validation dataset and generating images using various diffusion models and captions. The construction of this dataset is well described in Table. 6 The inference process straightforward: we measure CAS from images using four diffusion models (DP2.0, RVL44, EPIC and SD2.1) and feed these into a 3-Layer MLP model to generate the prediction. During training, the diffusion model is frozen, and only the MLP trained. For baseline in fake detection, we used the current state-of-the--ar, CLIP- VIT (L.C) (Ojha et al.) 2023), trained on the same dataset. shown Table. 7 this CAS ensemble MI.P model achieves an accuracy of 0.92 with only 500 training samples, surpassing the performance of CLIP-VI ViT (LC), which 0.87. is remarkable that the model performs well even though the diffusion models used for generating the test and training samples different. Moreover, as shown in Table g the performance of image generation source classification also reaches an accuracy of 0.90.
The characteristic of CAS measuring lower for out-of-distributinn samples in the diffusion model training distribution can applied to fake detection models and image generation source classifica- tion models.
14

---

Published as a conference paper at ICLR 2024
Model \a
0.1
0.01
0.001
0.0001
0.00001
SD2.1
1.240
0.237
0.033
0.020
0.070
DP2.0
0.305
0.053
0.005
0.008
0.198
Table 9: Normalized Root Mean Square Error (NRMSE) between log probability computed using
Song et al. 202Tb) and log probability computed using our method, across the Dreamlike Photoreal
2.0 (DP2.0) Art 2023 and Stable Diffusion 2.1 (SD2.1) (Rombach et al.
2022
10-2
10-3
10-1
10-5
10-6
10-7
Ratio
0.0
0.0
9.16
10-7
9.16
10-06
8.85
10
0.0018
Table 10: The proportion of each element of the estimated gradient being zero, various
Ground Truth
0=10-
0= 10
$20000
120000
120000
250000
100000
240000
Frequency
200000
80000
80000
160000
60000
60000
120000
80000
20000
20000
40000
0.000
0.025
0.050
0.075
0,100
0.000
0.025
0,950
0.075
10000
0.000
0.025
0,055
0.075
0.100
Estimated gradient
Figure Histogram of elements in estimated gradient for varying a.
A.3
ANALYSTS ON OPTIMAL FOR APPROXIMATION OF EQUATION
TO
In Sec. 4.4 provided the method to approximate Vxco(x, t)z, avoiding the resource- intennsiee
backpropagation process:
z Vxeg(x, TT cx + oz,t) cx t) 1 - (Eo oz,t) (x, t))
(11)
We set a to 10-3 which shows the best approximation performance in Table 15 However, there are two ambiguities: the optimal may vary for each diffusion model, and theoretically, smaller should yield better performance, which does not align with our experimental results. Therefore, in this section, explore finding the optimal and analyze the reasons behind its optimality. In quest to identify the optimal a for model performance, we extended our ablation study to include two additional models, SD2.1 and DP2.0. Our findings, detailed in Table reveal that value of 0.001 consistently yields the most accurate predictions of log probability across these models.
The core principle of differentiation suggests that approaches zero, the Normalized Root Mean Square Error (NRMSE) should correspondingly decrease. Contrary to this expectation, we observed an increase еггоr when was causes. This discrepancy can be attributed to the precision limitations of the float32 format. At very low values, subtraction operations lead significant rounding errors, nullifying the computed gradient. This phenomenon became apparent during our analysis of gradient calculations, where we observed noticeable increase in zero-valued gradient elements as approached zero. Our further analysis includes a comparison of the distribution of elements within the estimated gradient vector. We plot the histogram of elements the estimated gradient, particularly focusing on the proportion of elements that are zero in different scenarios. Fig. 7 and Table I0 illustrate these findings, showing a distinct rise in the frequency of elements near zero. Notably, while the distribution of estimated gradients at of estimated gradients at o = 10 7 is quite different from that the ground truth. The estimated gradients are concentrated near
10-7. This anomaly led to investigate the underlying
10 is relatively similar to that of the ground truth, the distribution 10-7
15

---

Published as a conference paper at ICLR 2024
Image Reward
CLIP Score
HPS
Pick Score
Ours
EER Pick Score dataset
0,419
0.383
0.306
0.278
0.376
Table 11: EER for each metric on our conducted Pick Score dataset.
Model
InstructPix2Pix
Canny Edge ControlNet
Scribble ControlNet
AudioLDM
p-value
1.74 10-8
3.18 10 10-18
2.16 10
3.22 10-7
Table 12: Statistical significance of human preference evaluation result for each modalities.
CAS score percentile
0~255% (Lowest)
75~100%
Model
25~~50%
50~75%
(Highest)
InstructPix2Pix
0.341
0.489
0.568
0.6
ControlNet (Canny Edge)
0.4875
0.4875
0.5
0.525
ControlNet (Scribble)
0.425
0.45
0.55
0.575
AudioLDM
0.4
0.439
0.575
0.585
Table 13: Histogram of average human scores divided based CAS score percentiles.
This issue is a recognized challenge in numerical differentiation and stems from the inherent preci- sion constraints of float32. Consequently, adopting a value of 10 3 a widely accepted compro- mise in addressing these computational limitations. Therefore, conclude that our setting of 10-3 is the promising selection.
ADDITIONAL EXPERIMENTS
B.1 EER METRIC ON PICKSCORE DATASET
We evaluated the Equal Error Rate (EER) of various methods, including proposed approach, using the Pickscore dataset. Since the task the Pick Score dataset slightly different from typical binary classification,
measured EER albeit with slight modification. For each image pair (image 1, image 2), the preferred image was assigned a score of I, and the less preferred one a score of O. We then calculated the prediction scores as the difference between the scores of the two images (i.e., score(image 1) score(image 2) for image 1, and vice versa for image 2). Using these predictions and the designated human preference scores, we computed the EER for each method, providing nuanced comparison of their performance. As shown in Table 11 EER measured by this scheme follows the behavior of accuracy.
B.2 ADDITIONAL RESULTS OF OTHER MODALITIES
Statistical Significance of Model Performance We conducted binomial test on the human evalu- ation results for multimodal dataset assess the probability of the models performing significantly better than random guessing, assumed to be 50% accuracy. The p-value, which measures the prob- ability of obtaining a result at least as extreme as the one observed under the assumption random guessing, was calculated for each modalities. As shown in Table [2] the p-values for all modalities are below 0.05. This indicates that the performance on each modalities is statistically significantly better than random chance, demonstrating the effectiveness of our method across various modalities.
Correlation Between CAS and Human Preferences In Sec. 5.2 we measured the performance of CAS across various modalities by measuring alignments with the preference of five participants for each test case. Based on these preferences, we analyze the correlation between CAS and mean human preference. Dividing the CAS scores into histogram intervals based percentiles, the corre-
16

---

Published as a conference paper at ICLR 2024
0.8
0.9
1.0
Accuracy
0.532
0.525
0.622
0.494
Table 14: Accuracy of our method on Pickscore dataset, varying X.
DDIM Inversion
2nd Order Recursive Inversion
Step
10
50
100
10
50
100
Accuracy 0.593
0.589
0.585
0.622
0.601
0.591
Time (s) 31.3
171.4
347.2
33.2
180.2
364.5
Table 15:
Accuracy in Pick Score dataset and average computation time spent for one sample with various settings of the inversion process.
sponding average human score is measured. As shown in Table 13 there is tendency for the mean human score to increase with higher CAS scores, indicating a correlation between CAS and human preference.
ABLATION STUDY
C.I PERFORMANCE CAS WITH CHOICE OF DIFFERENT
We conducted an ablation study of À term on CAS(x, the experiment setting of Scc we computed accuracy of our method on Pick Score dataset when A 0.8, 0.9, 1.0, and 1.1. As shown in Table 14 the accuracy is highest when À 1.0.
log log Po(x).). Following
C.2 PERFORMANCE CAS USING VARIOUS INVERSION PROCESS
The accuracy of CAS in Pick Score dataset provided in Sec. 5.1 is based the 2nd order recursive inversion with total step of 10. In this section, we provide the performance of CAS derived from inversions with different hyperparameters.
The results are presented in Table 15 From the standpoint of computation time, utilizing the 2nd order inversion is considerably more cheaper than simply increasing the steps in DDIM inversion. This is predominantly due to the fact that the time required to compute Ep(x) Ex t)z] vastly outweighs the inversion process itself. We can roughly infer the computation time based on the number of times the forward function of the diffusion model 0 is called for computing cg. If we denote the number of steps as n, for i-th order recursive inversion process, the forward function is called ni times. Meanwhile, to compute Ep(x) [z (xee(x, t)z], the forward function is called nk times where & is number of sampling of random variable z. This results in a total of n + k) calls. However, find that setting evident that increasing n. incurs a significantly greater computational overhead than increasing k, i.e. recursive inversion cheaper than simply increasing the number of inversion steps. This aligns with the results presented in Table 15
20 is optimal for stable experimental outcomes. Therefore, is
From an accuracy standpoint, we observe the following: 1) The 2nd order recursive inversion consis- tently outperforms the DDIM inversion, and 2) as the step size increases, accuracy diminishes. The superior accuracy of the 2nd order recursive inversion signifies that our methodology functioning anticipated. The decline in accuracy with increased step size in DDIM inversion suggests that merely enlarging the step size is insufficient for enhancing accuracy. In contrast, our approach manifests boost in accuracy.
17

---

Published as a conference paper at ICLR 2024
D THE DETAILED DERIVATIONS
D.I THE FULL DERIVATION OF CAS
In this section, we provide the detailed derivation of CAS in order. Please refer to Sec. and Sec. 4 for the definition symbols. Before deriving CAS, recap, DDIM (Song et al. 202 is defined as follows:
+ v1 at (x) t). (xce 1)
X Vo
(12)
v va:
where t = 1,2,,.,. ..., .N' and o is set 0 for all t in equation sample x0) can be calculated by:
To recap, the probability of generated
log po (x())) = log P1 (x(1)) + / Vx fo(x(1),
(13)
where probability flow ODE is given as follows: dx- = fo (x(), ()di
(Song et al. 2021b). The time
t is a continuous variable whose interval is defined as [0, 1] here.
The framework for calculating CAS is well illustrated in Fig. 3 which involves: 1) undergoing
DDIM inversion for the image and storing {Ee(Xe, c, t) (x 0, t, Xiy and x 2) Po applying equation 13 based on these values to compute Po and p c). and 3) calculating CAS((x, c, 9) log (x(c) log po (x). As mentioned above, equation 13 is calculated under the ODE of the diffusion process, which treats time as continuous, however, DDIM treats time as discrete. Therefore, DDIM should be rewritten in the form of ODE. As Liu al. (2022) did, replace discrete / - with a continuous version - 5 and subtract x, from both sides of the equation to get the differential form:
X
ea (x),)))
X1 8 X = (0+-8 8 aa)( Vael 0 Vat)
((lo)()-) + v(1 - ar)a
(14)
Then, dividing equation 14 by 8 and taking ó to zero, get the corresponding ODE:
dx
lim X,
X = at)
x(t)
(15)
di
5
2a(() 20(1)/1-(()
where x(t) and (t) are the continuous versions of (x, ) and (ox) M with time interval de- fined as [0,1]. Applying equation 15 to equation 13 we can compute the log probability DDIM inversion process:
Vx (a((1)
x(1)
2a(t)// l-(t) ))ar
E0(x(1),1)
log log =
(16)
2a(t)
Vx x(t)
xxxx(t)
Vx cx(x(tt)t)
log (x()) + a'(t)
dt a'tt)
2a(t) /1 - aa(t) dt
(17)
2a(()
(a()) D da J
a'(t) Vx ex(x(t),t) dt
=x0g)((x(1))
(18)
(a(0)
2a(t) /1 - -a(t)
va(O) D va(1)
[ d'(t) Vx (x()))))
(xgp)(x(1)) + log
dt,
(19)
2a(t) (o(t))/-o00 a(t)
where D is the dimension of image x0). By applying this formula, we can derive both log pe (x) and
log pe (x(c) from the values obtained through DDIM inversion, {<o(x,, )) feo t, 0) )
xyy and x When N is large, by transferring the time domain from [1, N] to [0, 1]. we can convert
xf and x into continuous variables with respect to as x" (t/N) and (t/N). Then, (0)
and (0) are derived as:
Va(0)
/ a'(t) V. C (x(xxx
log Pu Px(x(()) log + log
D va(I)
2a(t) dt, (20)
/ /a(0)
Vx co(x (t),
log (xx = log (1)(c) log
D
a'(t)
2a(t) - dt.
(21)
/ (a(1)
18

---

Published as a conference paper at ICLR 2024
Subtracting equation 20 from equation 21
CAS can be derived as:
Pi (xx (1))c)
L a' Vx co(x - Vx co(xx dt, (22)
CAS((x,c,) log
P1 (x (1)
2a(t)
where Vx ( c. and V 6 (1), 0) can be computed by Skilling- trace
estimator (Skilingg 1989; Hutchinson 1989). Vx Eg (x()).)) = px))) Vx (x, t)z] with respect
to our approxımation method introduced equation I0 With the obtained values from the Skilling- Hutchinson estimator, the numerical integration method is applied to obtain CAS.
D.2 DETAILED EXPLANATION OF DDIM RECURSIVE INVERSION
In this section, we provide the underlying idea of the DDIM recursive inversion. To recap, the ideal DDIM inversion is given follows:
X, Gp 1 + 1E (X,
(23)
where 0:-1 you to use itself to recover T2. Therefore, is convention to approximate it by:
(VI ai
Cte az), and oe = 0. However, it is contradictory
X = a 1,-1 + (x)
(24)
Our DDIM recursive inversion is defined as follows:
1 + be 16-160(() (x)-1,1) ,()
(25)
for 2,3, and x] Xx, and we denote the approximation x; Rs X7 as the n-th order recursive inversion.
To effectively illustrate underlying idea of this DDIM resursive inversion, we consider the simple
case that: 1) xx the latent derived by DDIM inversion is closer to true latent than X;-1, i.e.
|[x-Xx, < [x -x- and 2) for two random vectors u and v close to x,, if [xx-u] < |
then ||o [x. (u, t)ll [leo(x,, co(v, t)ll holds. Subtracting the ideal DDIM inversion
(equation 23) from 2nd order DDIM inversion (x? in equation 25 the following equations can be
derived:
|| x.ll = b, (l 60 .)1
< b, || (x:-111 - (xe: 8)ll
||: xcll
(26)
by the assumptions. Assuming that ||x| 1 x|| < ||x - x||. the following equations can be derived:
| - x,|| = b b { 1- ,t) - ex(x;,t) t)l
< bb-1llee x 2,t) x),
-| X;
(27)
expect that the error between the true latent and predicted latent decreases, i.e., || X, > ||*? - xt > || x;ll > ... holds. Note
By applying equation 27 recursively equation 26
that we multiple assumptions for simplicity, which effectively illustrates the underlying idea of the DDIM recursive inversion.
RANKING EXAMPLES OF CAS
We have prepared several examples demonstrating the ranking of CAS across various modalities, including text-to-image, image-and-instruction to image, edge-and-prompt to image, and scribble- and-prompt image.
19

---

Published as a conference paper at ICLR 2024
Condition
High rank
Ranked Images
"A winter wonderland night, with ice sculptures glowing under the aurora borealis, people skating on frozen lake, and cozy igloos serving warm, spiced drinks."
Figure Texe-to-image. Stable Diffusion v1.5 (Robaach al. 2022 is leveraged.
Condition
High rank
Ranked Images
"A serene beach at sumrise, with gentle waves, clear sky, and a lone figure practicing yoga on the sand, with distant boats the horizon."
Figure 9: Text-to-image. Stable Diffusion v1.5 Rombach et al. 2022) is leveraged.
Condition
High rank
Ranked Images
"Make it space suit"
Figure 10: (Image, instruction) (estructioo)-to-imaaeee to-imaee. InstructPix2xPix Brooks et 2023) is leveraged.
20

---

Published as a conference paper at ICLR 2024
Condition
High rank
Ranked Images
"Add herd of elephants"
Figure 11: (Image, InstructPix2xXix
Brooks ct al. 2023) is leveraged.
Condition
High
Ranked Images
"Digital Art- Superman by Kai Saarto"
Figure 12: (edge, coompt)-to-image Controlnet (Canny Edge) Zhang et al. 2023 is leveraged.
Condition
High rank
Ranked Images
"Log cabin in the woods with giant spider painting Winter Scenes"
Figure 13: (edge, prompt] - tmmge. Controlnet (Canny Edge) ZZhang et al. 2023 leveraged.
21

---

Published as a conference paper at ICLR 2024
Condition
High rank
Ranked Images
"South Island Braided Rivers"
Figure 14: (scribble, -to-iaee Controlnet (Scribble)
(Zhang et al. 2023 is leveraged.
Condition
High rank
Ranked Images
"Alps Photograph- Early Winter Mood by Nicolas Schumacher"
Figure 15: (scribble, prompt)-to- tto-image. Controlnet (Scribble) (Zhang et al. 2023 is leveraged.
22


Reviews:
[{"review_id": "Zk9aeilLvs", "reviewer": "Reviewer_8WfE", "summary": "This paper introduces a novel methodology to calculate a condition alignment score (CAS), which is both training-free and applicable across various modalities. The proposed approach involves calculating the ranking metric $\\log p_\\theta(x|c) - \\log p_\\theta(x)$ to serve as the alignment score. The authors leverage the probability estimation from flow ODEs to determine the requisite probabilities (with a trained diffusion model (DDIM)). Empirical findings support the efficacy of the CAS method.", "questions": "- Could you clarify how the probabilities in Figure 2 were calculated? Are they derived using Equation 4 in conjunction with the Skilling-Hutchinson trace estimator?\n- Has the DDIM Recursive Inversion been introduced in previous literature?\n- The poor performance of all baselines on the Van Gogh dataset is unexpected, especially considering that the sample in Figure 5 appears distinguishable. Could you provide an explanation for this phenomenon?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The CAS method's universality across different modalities and its training-free nature are significant advantages.\n- The proposed framework is principled and reasonable.", "weaknesses": "- This paper can be improved by adding more essential technical details. For example,\n  - This paper follows many prior works on diffusion model, so there are a lot of citations. When referring to a cited paper, it's unclear for the readers what the paper is about. Maybe a small summary or referring by their method names will be better.\n  - The captions can be improved. For example, consider adding a detailed description for the piepline of the method in Figure 3. Explain the full name and formula of NRMSE in Table 1.\n  - More ranking examples can be shown in Appendix. \n- The experiments can be improved in those aspects:\n  - Report the standard deviation for experiments. Each prompt used in Van Gogh Dataset can be used to generated more groups of test samples to test the baselines. \n  - The numbers of test samples in different experiments are small. As the accuracy reported in Table 3 is near to 50%, it's hard to judge whether this method works well.\n  - The ablation on $\\lambda$ for the ranking metric of CAS is missing. \n- It will be insightful to show some failure cases and analyze the failure mode to help understand the limitations of CAS.\n- There are some typos, for example, the caption for Table 1 is ended without any punctuation.\n\nThis paper can be greatly improved after a careful revision. The writing for the current version is a bit rough and obscure. Many details should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ZhvoSEgDy", "reviewer": "Reviewer_uTB1", "summary": "In this paper authors design a novel metric to decide whether result of conditional sampling from diffusion model should be rejected or accepted. Noting that typically user is presented a grid of images and user has to do the wotk himself. Obviously, it would be beneficial to automatize this process. The key idea is to accurately compute difference of unconditional  and conditional log probability. And the key difference to literature is that no curated dataset is needed and separately trained  scoring model.", "questions": "- About Table 1, I wonder why RMSE is biggest at the edges (\\sigma = 01, and \\sigma = 0.00001) ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "I really liked this paper, it presents an interesting problem and a narrative on how the problemd was solved.  And the idea also makes sense, how to improve the log p(x) computation is reasonable, I also liked solution to compute the difference in N(0,I) and then correct the difference via computing the path.", "weaknesses": "Two biggest issues to me are in empirical validation and lack of complete derivations of the main results. \n- Main issue with the empirical results is that authors report accuracies. But the accuracy is notoriously bad measure for binary classification case, such as this (keep image or rejecct it). Reasonable metrics are for example, equal error rate (EER) or AUROC. I personally like EER. EER can be nicely computed for example with (https://github.com/bsxfan/PYLLR). Interestingly, same package also allows to plot DET plot that shows the whole ROC in a bit more meaningful way. \n- I think it i would be especially useful to have the complete derivations of the main result in the appendix. \n- Eq. (9)  also needs a bit more explanation that could be written in the Appendix. \n\n- Some notatioanl issues, such as vectors are marked with bold symbol and non-bold symbol. \n- What is the intuition that in Fig 2, - log p(x) shows better performance than log p(x)? \n- How was ground truth estimated from Van Gogh dataset? In Pick Score dataset I see that human annotation is used. Did you annotate yourself Van Gogh dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v0XmLBqfsi", "reviewer": "Reviewer_cbcH", "summary": "This paper proposes a simple metric for measuring alignment between the generated sample and the condition for conditional diffusion models (such as text-to-image or text-to-audio models). The authors propose to leverage the conditional likelihood of the model which can be computed through the diffusion process, and name this CAS — condition alignment score. They also propose a method to improve the DDIM inversion that is required to compute the likelihood, and suggest a simple approximation to speed up this computation. CAS is measured for several domains, and found to align well with objective metrics like CLIP score or subjective human evaluations.", "questions": "Will the dataset of human evaluations (used in Section 5.2) released along with the code?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The problem of defining a universal metric for condition alignment is important but under-researched. Using the likelihood computed from the model itself is a clever choice, and is analogous to intrinsic measures like perplexity in language models.\n\n2. The paper is well written and easy to understand, and the motivation for using the metric is explained clearly. The contributions for improving and speeding up DDIM inversion are also described well.\n\n3. Evaluations are conducted across several domains including image and audio, and CAS shows strong correlation with previously used measures like CLIP score. This is useful for researchers using diffusion models in other domains.\n\n4. Most importantly, the authors have mentioned that the code for computing CAS will be made publicly available. This should make it easy to use as a metric for future work on conditional diffusion models.", "weaknesses": "1. In Section 2 (paragraph 2), the authors mention that CAS score relies on computing the likelihood of samples generated from a conditional diffusion model, for which they use a method proposed previously by Song et al. This was later used by Zimmermann et al. to measure the class-conditional likelihood of images. The key difference, as described in Section 4.1, is that the authors subtract the unconditional likelihood. While the authors provide some empirical justification for this (Figure 2), it may be useful to conduct a more thorough investigation to justify the hypotheses proposed in Section 4.1, for example by training conditional diffusion models on some toy data.\n\n2. In Section 4.4, the approximation only holds when $\\sigma$ approaches 0. Based on this, we should expect the accuracy for the approximation to improve as we choose $\\sigma$ closer to 0, but this is not true in Table 1. Do the authors have some explanation for this behavior? Further, given the large range of NRMSE values, how should practitioners choose this parameter when computing CAS for a new domain?\n\n3. In Section 5.2, the authors only use a single metric (accuracy) to measure the correspondence between CAS and human preference. However, this does not give any insights on how well CAS difference correlates with human judgment. It may be more useful to show a plot with CAS difference between the samples and the corresponding human preference accuracy (for e.g., through a histogram), where we expect the accuracy to increase as the CAS difference gets larger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)