SelfEval: Leveraging discriminative nature of generative models for evaluation
Authors: Sai Saketh Rambhatla
Ishan Misra
Venue: Accepted by TMLR ()

Abstract: We present an automated way to evaluate the text alignment of text-to-image generative diffusion models using standard image-text recognition datasets. Our method, called SelfEval, uses the generative model to compute the likelihood of real images given text prompts, and the likelihood can be used to perform recognition tasks with the generative model. We evaluate generative models on standard datasets created for multimodal text-image discriminative learning and assess fine-grained aspects of their performance: attribute binding, color recognition, counting, shape recognition, spatial understanding. Existing automated metrics rely on an external pretrained model like CLIP (VLMs) or LLMs, and are sensitive to the exact pretrained model and its limitations. SelfEval sidesteps these issues, and to the best of our knowledge, is the first automated metric to show a high degree of agreement for measuring text-faithfulness with the gold-standard human evaluations across multiple generative models, benchmarks and evaluation metrics.  SelfEval also reveals that generative models showcase competitive recognition performance on challenging tasks such as Winoground image-score compared to discriminative models. We hope SelfEval enables easy and reliable automated evaluation for diffusion models.

Published in Transactions on Machine Learning Research (12/2024)
SelfEval: Leveraging discriminative nature of generative models for evaluation
Sai Saketh Rambhatla GenAl, Meta
rssaketh@@meta.o com
Ishan Misra
imisru@mella. com
GenAT, Meta
Reviewed on OpenReview: https: // openreveew. net/ per/form?? id- 0mGho8uru5
Abstract
We present automated way to evaluate the text alignment of text-to-image generative diffusion models using standard image-text recognition datasets. Our method, called SELFEVAL, uses the generative model to compute the likelihood of real images given text prompts, and the likelihood can be used to perform recognition tasks with the generative model. We evaluate generative models on standard datasets created for multimodal text-image discriminative learning and assess fine-grained aspects their performance: attribute binding, color rocognition, counting, shape recognition, spatial understanding. Existing automated metrics rely an external pretrained model like CLIP (VLMs) LLMs, and are sensitive to the exact pretrained model and its limitations. SELFEVAL sidesteps these issucs, and to the best of our knowledge, is the first aulomated metric to show a high degree of agreement for measuring text-faithfulness with the gold-standard human evaluations across multiple generative models, benchmarks and evaluation metrics. SELFEVAL also reveals that generative models showcase competitive recognition performance on challenging tasks such as Winoground image-score compared to discriminative models. We hope SELFEVAL enables easy and reliable automated evaluation for diffusion models.
1 Introduction
In the past few years, generative image models have rapidly advanced and state-of-the-art text-to-image models now generate high quality realistic images. While lot of research effort focussed on improving these models, their evaluation has received considerably less attention. Evaluations for lext-to-image models typically focus on two aspects: (1) quality of the generated image; and (2) the aligmment between the generated image and the input text, i.e., the "faithfulness" of the generation. The gold standard for evaluating text-to-image models is to compare generations from pairs of models using human judgement. However, using pairwise human evaluations does not scale to lots of models generations, making it difficult to convert them to ordinal metrics to rank models. Thus, automatic evaluations are commonly used as a proxy for comparing models.
In this work, we focus on automatic evaluations that measure the text adhering capabilities of a generative diffusion model and ask the question: can the diffusion model itself be used measure the relatedness of image-text pair and thus evaluate its own generations? Most works on text-to-image diffusion models focus sampling good images given text prompt. However, shown in Figure 1, diffusion models can be used to estimate the conditional likelihood of an image x given a text prompt c, i.e., px)().. We propose SELFEVAL which is a practical way to estimate such likelihoods accounting

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Standard Diffusion sampling
Self Eval
Self elassification
xT
Noise
Zgi Real Input
Xo: Real Input
pioalca) p(ule) :
K
V
K v
K V
K v
v
K
p(xolc)
Q
Q
Q
Q
Q
((
dog
dog
Co dog
Cy cut
Cyp
(Left) Starting from noised input, the standard diffusion sampling method denoises the imput iteratively to generate images from the input distribution. (Middle): SellEvaL takes pair (image xo and conditioning iterative fashion. (Right): Given an image, Xo and n captions, (Co, Ci,... .I,...,..), SELFEVAL a principled way convert generative models into discriminative models. We show that the classification performance of these classifiers can be used to evaluate the generative capabilities.
Figure 1: Illustration of proposed method:
to estimate the likelihood p(xa (e) of the pair in
for numerical issues arising in standard diffusion models. We show that these likelihoods can be used directly to solve recognition tasks and evaluate the model's text-faithfulness ability.
SELFEVAL repurposes standard multimodal image-text datasets such Visual Genome, COCO and CLEVR to measure the model's text understanding capabilities. SELFEVAL uses ground truth (real) image-text pairs for evaluation and computes classification accuracy making more robust and interpretable. Our evaluation allows us to assess fine-graine siee-graindd aspects such as the model's ability to recognize colors, count objects etc. We apply our method to a wide variety of diffusion models: different types of image representations (pixel based, latent space based), different text encoders and model sizes. SELFEVAL's automatic evaluation results are in agreement with the 'gold-standard' human judgements making SEL.EVAL suitable for evaluation.
Existing automated evaluations for text faithfulness of generative models rely on an external discrimina- tive model, e.g., CLIP or LLMs, to measure the 'relatedness' the generated image to the imput text. As we show in Figure 2, relying on an external model leads to three major issues. First, the automated metrics vary grcatly depending on the type of the external model used for evaluation and they often have an arbitrary range, e.g., See Table 2,3 for the range of CLIP Radford et al. (2021) and MID Kim et al. (2022) scores. Second, many generative models rely on external model such CLIP's text encoding during training, and thus using the same CLIP model for automated evaluation biases the results. Finally, the external model itself can have several limitations (like poor performance on few image-text tasks for CL.P and hallucination for LLMs Xu et al. (2024)) making its scores unreliable.
SELFEVAL only uses the generative model and thus its scores directly reflect the strengths and weaknesses of the generative model. Note that most automated metrics operate on generated images whereas SELFEVAL uses real images from image-text recognition datasets.
2 Related Works
Generative models learn to model the joint distribution, p(X, Y) of data consisting of an observed variable X and the target Y. The model can then be used to sample novel data. In this work, we are interested in image generation models.
Generative models:
Generative Adverserial Networks (GAN) Goodfellow al. (2014); Radford et al. (2015), Variational Au- toEneoders (VAE) Kingma & Welling (2014) and Denoising Diffusion Probabilistic models (DDPM) Ho et al. (2020) are some of the most popular image generation models in the literature. GANs belong to the category of generative models, where two distinct components, generator and discriminator, are pitted against each other within a zero-sum game framework. VAEs are a category of autoencoders

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Counting objects
Spatial awareness
Binding attributes
Text corruption
LL (xT-414)
remuse LEN-LL
304
STOP
Ieee sheep sndll a dog Four sheep and deg
Stupk Janne Straet digs niret to stop sign Siloed apple and white rice
Welle and dlool Hoc
Bool of publing es plate
Plane band paddiing on
Figure 2: Drawbacks of CLIP for generative model evaluation. (Left) We compare the CLIP similarity scores of two latent diffusion models trained with CLIP VIT-L/14 (LDM-CLIP (VIT-L/14)) and OpenCLIP VIT- H/14 (LDM-CLIP (VIT-H/14)) text encoders. On the left, we compare the CLIP similarity scores, computed using CLIP ViT-L/M, on prompts from DrawBench, Winoground and, COCO datasets. The plot on the right compares the CLIP similarity scores computed using OpenCLIP ViT-H/11 model. The ranking changes depending on the model used. (Right) CLIP has poor performance in tasks of counting, spatial relationships, binding attributes and text corruption which constitute about 25% prompts in DrawBench. In each example, the correct caption is shown in green and CLIP picked the caption in bold. Using CLIP to evaluate text to image models on such prompls is not optimal.
that ensure "regularity" within the latent space by constraining their distribution to closely align with a well-behaved and typically standard normal distribution. In more recent times, DDPMs have exceeded the capabilities of all preceding tatte-oftthe-art image generative models in terms of their generative prowess. Drawing inspiration from non-equilibrium statistical physics, Diffusion probabilistic models Sohl-Dickstein et al. (2015) employ a forward diffusion process to gradually destroy the structure in the unknown input distribution and transforming it into a well-behaved and tractable distribution. A reverse diffusion process is trained to learn to restore the structure, thereby learning the input distribution. An explicit connection between diffusion models and denoising score matching in Ho et al. (2020), enabled a simplified objective for training diffusion models. We utilize diffusion models in this study due to their outstanding image generation performance Dhariwal & Nichol (2021).
Diffusion models: In a relatively short time, diffusion models have surpassed GANs and VAEs as the defacto models for image generation due to their superior quality Dhariwal & Nichol (2021) and fiexibility. Numerous studies have shown that diffusion models can be conditioned on a variety of modalities, including object classes Peebles & Xie (2023); Ho et al. (2020), natural language captions Saharia et al. (2022); Rombach et al. (2022); Nichol et al. (2022); Ramesh et al. (2022), camera pose Liu et al. (2023), images Brooks al. (2023), bounding boxes Li al. (2023b), segmentation, edge and depth maps Zhang & Agrawala (2023). Among these, text-conditioned diffusion models have attracted siguificant interest and popularity. Given paired image, caption data, these models are trained to fuse the caption features, extracted using a pre-trained text encoder, with the image features using cross attention. Text-to-Image (T2I) models demonstrate a remarkable comprehension of compositionality within text, often highlighted by their capacity to generate images based on counterfactual textual descriptions (like avacado shaped chair elc.). The most popular text encoders in use loday are text encoders from the CLIP Radford et al. (2021) and the T5 Raffel et al. (2020) transformer. In this work, we analyze the text understanding of the diffusion models trained with diffcrent text encoders.
There exist two families of diffusion models in the literature, namely, pixel (PDM) Saharia et al. (2022); Ramesh et al. (2022) and latent diffusion (LDM) Rombach et al. (2022), differing primarily in the nature of input. The diffusion process in pixel diffusion is performed on pixels making these models computationally expensive. LDMs Rombach et al. (2022) operate on the autoencoder's latent space, balancing the computational constraints with the quality of PDMs. In this work, we analyze the text understanding of two state-of-the-art models with different text encoders from PDMs and LDMs.
Classifiers with diffusion models: Lately, there has been increase in the usage of conditional diffusion models as classifiers, driven by their superior understanding of the conditioned modality. These models are surprisingly good at capturing intricate patterns and dependencies within the conditioning

---

Published in Transactions on Machine Learning Rescarch (12/2024)
imput, making them strong discriminative models across range of downstream tasks. Notable works include He et al. (2023), Mukhopadhyyy et al. (2023) that either finetune diffusion model, linear probing, for several classification and reasoning tasks. Unlike these methods, we do not train any models but instead convert the generative model into a discriminative one to understand its text understanding capabilities. Along similar lines to SELFEVAL, Clark Jaini (2023); Li al. (2023a) employ the ELBO loss as proxy to estimate the likelihood scores (and subsequently the posterior using Bayes' rule) from diffusion models for several image understanding tasks. Li et al. (2023a)also report promising results ITM tasks the Winoground Thrush et al. (2022) dataset. Instead, we propose a systematic way, accounting for numerical stability, to estimate the likelihood of an image given the text from a conditioned diffusion model. To the best of our knowledge, SELFEVAL is the first method to show that the discriminative performance of genorative models aligns closcly with goold-standard' human evaluations in the assessment of generative models.
Image-Text Text Matching for evaluating generative models: CLIP R-procision Park al. (2021) is metric evaluate the text understanding capabilities of generative models similar to SELFEVAL. While CLIP R-precision measures the text retrieval performance given generated image, SELFEVAL uses the ground truth image-text pairs for evaluation. The generated images are often ont-of-distributioo to the external model, i.e. CLIP, making the score unreliable. SELFEVAL avoids this by using the generalive model, instead of an external model, for evaluation.
3 Method: Converting generative models to discriminative models
Our method converts generative (diffusion) models into discriminative models by simply changing the inference, and does not require any retraining. This allows us to use the diffusion model itself on a variety of different imago-text benchmarks and assess the diffusion modcl's image-text understanding capabilities. We briefly discuss an overview of diffusion models in Sec. 3.1 followed by our proposed method in Sec. 3.2
3.1 Preliminaries
Diffusion Probabilistic Models (DPM) Sohl-Dickstein et al. (2015) belong a class of generative models trained to 'denoise' inputs created by Markovian forward process. The forward procoss starts with sample Xo and repealedly adds Gaussian noise over L timesteps to generate X,;
((x:|x=1) N(x:; V1 - 31x,
(1)
Here ((xg) the data distribution. 3g is the strength of the noise at timestep & with 90 = 0, = 1. Note that X₃ are the same size as the input. The joint distribution of the input along with the latents q( ((XoT)
T
((xo:T) = ((x)) II [xxx
(2)
To sample images, one applies the reverse process, p(x;-1/x)), starting with xT sampled from the unit. normal distribution, So the joint distribution of the reverse process be described as
p(xer) p(xx) II ((x
t=1
The reverse procoss p(x{-1/x:) is not tractable and is often modeled using neural network whose parameters are characterized by 0, i.e. pa(x--1|x, ~ (x
po (x., (), Eg (x.x, ())()..

---

Published in Transactions on Machine Learning Research (12/2024)
3.2 Likelihood estimates from diffusion models
We specifically focus text-to-image diffusion models, although formulation extends to any conditional diffusion model. Text-to-image diffusion models are trained on large datasets of image-text (x, c) pairs and model the reverse diffusion process p(x+-1 Xx, c). We 'invert' such a generative model and use it to estimate the likelihood of a real image x given a text caption c, i.e. p(x(c). We note that our method only changes the inference a diffusion model and does not require any training. Assuming uniform prior on the classes, the likelihood p(x|c) can be converted into the posterior, p(c/x) using Bayes' rule, i.c. p(c]x) p(x|c) where C is the set all classes.
Tel
Given the reverse process of a diffusion model parameterized by 0, the likelihood for datapoint X is
po c)
= po (e(xxrl()dxx11))
(4)
T
=
p(XT)
II
po (x (x c)dxx
(5)
t=1
Since the diffusion models reverse process p-)-) is also a gaussian, we can further write this as
p(xo c) p(xT)
v (2x) Egl
x exp
xx-1 (x( c))² Exi(xx-1
(x1-1 - (o(x). c)) dx
(6)
Here, p(XT) ((,,,) 1). For the sake simplicity, we denote any realization of the random variable x as Xg. Given natural language caption c, an image Xy and the noised latents 21.7, the quantity inside the integral in Eq. 6 can be estimated numerically. We compute Monte Carlo estimate of the integral by sampling N noise terms (c) and computing p(xple)
T p(x]lc) = Σ p(x) II p(xx"_1>>x c)
N
n=1
1=1
where
v Bix²_1 + Bar""
Practical considerations: The terms on the RHS of Eq. are multivariate gaussians and analytically computing them involves exponentials which can be numerically unstable. This can be prevented by computing log probabilities instead. Taking log both sides of Eq. 7, we get
N
log logp(xolc) = log D p (x}
Il p( [xx
(8)
N (log p(x)))
T
1ogp(x"__lx,;
(9)
R=1
Where Eq. 9 follows from Jensen's inequality for concave functions, i.c E(f()) < f(())). All terms in Eq. 9 are analytically computable log probabilities of multivariate gaussians, offering numerical stability.

---

Published in Transactions on Machine Learning Research (12/2024)
Attribute binding
Color
Count
Shape
Spatial
Text corruption
blue lake and browa
orunges in black bowl lake blue
sphere
red sphere
Swan front of boat.
wooden lined are spoons
oranges green
Three
And
under woodan spoons are
Figure 3: Representative samples from the benchmark. We divide the evaluation into six broad tasks, namely Attribute binding, Color, Count, Shape, Spatial, and Text Corruption. Each task is designed to evaluate a specific aspect of text laithfulness mimicing the categories in DrawBench. Each task is posed as an image-text matching problem, where given image, the goal is to pick the right caption among distractors. The figure above shows examples from each task with the right caption highlighted in green.
Implementation in practice: Given an image x and a caption c, we estimate p(x(c) as follows. As shown Fig. 1, the noised latents X1. are computed using the forward process (Eq. 1). The final latent XT is denoised for T' steps using the reverse process (the neural net) to obtain the denoised latents -1:T-1- This process is repeated for N independent noise vectors resulting in {xT=r) n=1. (xi-r-1)a J.Y Finally, the likelihood can be computed as p(x(c) = =E p(x7) I[ p(x]_1xx) c) (Eq. 9). We now show how estimating p(xolc) allows us to evaluate diffusson models
3.3 SelfEval evaluate diffusion model's text faithfulness
The text faithfulness of diffusion model measures its ability understand the text prompt and ground it in the generated image output. The 'standard' way of evaluating text faithfulness uses manually curated list of text prompts to generate images. The 'alignment' between the generated images and the text prompts can be measured using external model human evaluator. The text faithfulness of a generative model inherently also measures its vision-language reasoning abilities. Thus, SELFEVAL, we propose to directly measure the generative modcl's vision- languaee discriminative performance as a way to evaluate its text faithfulness.
We pose the SELFEVAL evaluation an image-text matching problem and measure the generative model's recognition performance standard discriminative image-text datasets. Thus, SELFEVAL does not rely on external models such as CLIP, does not need human evaluators, and does not need manual text prompt-sct curation.
Image-text matching problems, such as image classification or retrieval, can be reformulated as picking the correct caption for a single image x from a set of captions {c.). First, we can use a diffusion model estimate p(x(c;) (described Sec. 3.2) for each of the captions. The likelihood, p(x(c;), is then converted to the posterior, p(c; x), using Bayes' rule. Finally, the caption with the highest likelihood, i.e., arg max,, p(c, is chosen as the right one.
To compute the SELPEVAL score, we construct an evaluation set from paired (image, caption) datasct. Each sample in this set consists of an image, its correct caption, and several distractor captions. We ensure that the distractor captions are sufficiently hard to increase task difficulty. For each image- caption set, we calculate the posterior probability using SELFEVAL and determine the model's accuracy, i.e. SELFEVAL score, over the entire set with the ground truth.
4 Experiments
We now use SELFEVAL to evaluate text-to-image diffusion models. In 4.1, we introduce our benchmark datasets and models, and present the SELFEVAL results in Section 4.2.

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Juman eostuntore of Plael difuccn mozels
Comperison ofP Medea Leing Settial ranga rowo
Human untionn of Letent diaatos mode
Compenisor of Mocelc udig TEvs Lakou (an
3.5
677
IONCUP 401
LINGL L041
1.43
335
Count
Spetu
Atrbute
Caler
Shace
Spara
Figure 4: Evaluating text-to-image models using human evaluations and SELFEVAL. We evaluate different types of text-to-image models such pixel diffusion (first two columns) and latent diffusion model (last two columns), and models that use different text encoders such as T5 XXL and CLIP. We observe that across all 4 diffusion models the relative ordering given by SELFEVAL's accuracy correlates with the pairwise human evaluation results. We also observe that latent diffusion models have a higher SELFEVAL accuracy than pixel diffusion models suggesting better text-faithfuluess. Using the stronger T5 text encoder leads to better performance across human evaluations and SELFEVALL
4.1 Benchmark and Evaluation
In SELFÉVAL, we pose the text faithfulness evaluation as an image-text matching task, where the goal is to pick the right image caption pair among distractors.
Tasks. We identify six broad reasoning tasks for evaluation (Figure 3): 1) Attribute binding, 2) Color, 3) Count, 4) Shape, 5) Spatial relationships, and 6) Text corruption. Each of these lasks evaluate the model's understanding of a specific aspect of text laithfulness and is similar to the categories of prompts from DrawBench Saharia et al. (2022). The six tasks are constructed using data from TIFA Hu et al. (2023), CLEVR Johnson et al. (2016) and ARO Yuksckgonul al. (2023).
Datasets. TIFA Hu et al. (2023) consists of 4000 text prompts, collected manually and from image captioning datasets, to evaluate the text faithfulness of generative models. In our evaluation, use ~2000 of these text-prompts that are constructed from the COCO Lin et al. (2014) dataset and convert the dataset from question-answen anwweriisw to an image-text matching format as detailed in the supplement. Attribution, Relation and Order (ARO) Yuksekgonul et al. (2023) is a benchmark that uses data from Visual Genome Krishna et al. (2017) for attribute and spatial relations, and COCO for ordering tasks. CLEVR Johnson et al. (2016) is a benchmark for compositional understanding and visual reasoning using synthetic images. We adopt the splits proposed by Lewis al. (2022) for our case.
We divide the datasets among all the reasoning task follows. For attribute binding, we combine samples from ARO (attribution) and CLEVR. For colors and counts, we use corresponding samples from TIFA and CLEVR. For shapes, we use samples from CLEVR. Data for spatial relationships is from TIFA, CLEVR and ARO (relations). The data for the text corruption task is from the ARO (order sensitivity) dataset. sample of each task consists an image and multiple text prompts and the performance on the task is the accuracy of pairing the image with the right caption.
We measure the performance of text-to-image generative models on the benchmark using the following evaluation methods.
SelfEval (Ours) is an automatic evaluation method and uses both the images and text from our benchmark introduced in Scction 4.1. For cach benchmark task, we randomly sample 1000 examples and evaluate the classification performance on them. We repeat this three times and the report the mean accuracy. We use 10 trials (L.e. N 10) and perform diffusion for 100 steps (i.e. T 100) for all the models. Refer to the supplement for ablation experiments on N, T.
Human evaluations are the gold standard for judging the performance of poct-to-image models using pairwise comparsions. We present humans with generations from two models and ask them to vote for one of four choices: "both" the generations are faithful, "none" of them are faithful, or if only one of

---

Published in Transactions on Machine Learning Rescarch (12/2024)
the two images ("Image 1" or "Image 2*) demonstrates fidelity to the given prompt. For simplicity, we only report votes where there is clear preference for model. We randomly pick 250 text prompts from each benchmark task as conditioning for human evaluation and the images are generated using DDIM Song et al. (2021) sampling, with 100 denoising steps. Note that unlike SELFEVAL, human evaluations do not use the real images from the benchmark tasks and the human evaluators only look the generated images.
4.1.1 Models
We use models with different image representations: pixel diffusion models which directly use the pixel RGB values, and latent diffusion models where the image is projected into a latent space using an auto-encoder. We pick models trained with different text encoders within each class. This enables us analyze the effect of text encoder on the final performance within each class,
Diffusion models with CLIP text encoder.
We employ model trained with the OpenCLIP Thharco et al. (2021) text encoder with ViT-H/14
backbone for latent diffusion, accessed via an API containing open-sourced model weights. This model, trained on a public dataset of billion images (cxcluding explicit matcrial), outputs images of 512 x 512 resolution. For pixel diffusion, we use the architecture of DALL-E-2 Ramesh et al. (2022) in our experiments and train model. This model uses a CLIP (ViT-I//14) text encoder, produces images of 64 x 64 resolution, and has a total of 4.2B parameters. It is trained for 2M steps on an internal image-text dataset (Internal-Dataset)
Diffusion models with T5 text encoder. We train a UNet model for latent diffusion, similar to Rombach et al. (2022), but with the CLIP text encoder replaced by a T5 XXL Raffel et al. (2020) text encoder. This model outputs images of 256 x 256 resolution. Trained on Internal-Datasel for 2M steps using a latent space with 4x downsampling factor, the model has a total of 5.8B parameters. We train pixel diffusion model with 7.5B paramcters, similar to Imagen Saharia et al. (2022), on 64 64 resolution inputs for 2M steps using the same data. Following this, we use super-resolution model to upsample the output to 512 x 512. With the exception of the CLIP-based latent diffusion model Rombach et al. (2022), all the other models are trained for the same number of steps the exact same data to ensure fair comparison.
4.2 Main results
We evaluate the four text-to-image models and report results in Figure 4. For SELFÉVAL, we report the accuracy difference with the chance accuracy, since each of the lasks has different degree of difficulty.
Agreement between SelfEval and human evaluation. In Figure 4, we evaluate four different diffusion models using both human evaluation and SELFEVAL. The human evaluation performance, measured via pairwise comparison, aligus with the ranking given by SELFEVAL for both pixel and latent diffusion models. To our knowledge, this is the first work to correlate the discriminative performance of generative models with human evaluation for text-to-image diffusion models across various modcls and lasks. The strong alignment between SELFEVAL and human raters suggests that SELF EVAL is a reliable and interpretable way for evaluating and comparing the text faithfulness of different diffusion models.
Next, we use SELFEVAL to further analyze the performance of diffusion modcls.
Effect of the text encoder. Comparing the different. text-encoders used Figure 4, we observe that diffuuson models using the stronger T5 text encoder perform better on most tasks than the ones using the CLIP text encoder. The stronger performance of Tasbased models holds for both human evaluations and SELFEVAL. The SELFEVAL results indicate that diffusion models using the Case-based encoders perform poorly on the Count task, even worse than random chance. For the Text Corruption task, which involves identifying a linguistically correct sentence among distractors with shuffled word order,

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Comparison of LDM VS PDM
Diffusion models evaluated on the Winoground dataset. We measure the image score (ac- curacy of picking the correct image given a text prompt) and text score (aceuracy of picking the correct text given an image). Using SELFEVAL allows us to use diffusion models for both tasks unlike Li et al. (2023a) which leads to zero image score.
Table 1:
0.4
LDM
PDM
of vates
0.13
Method
Model
Image Score
Text Score
CLIP (VIT-L/14) OCLIP (VIT-H/14)
8.00
30.25
12.75
30.75
0.0
Human evaluation
SeifEval
et al. (2023a)
LDM-CLIP
34.00
Figure 5: Pixel vs Latent diffusion. Hu- man raters rank the generation of latent mod- els higher than pixel models in text faithful- ness. SELFEVAL exhibits similar trend.
SELFEVAL
LDM-CLIP
7.25
22.75
SELFEVAL
PDM-CLIP
14.00
17.00
SELFEVAL
PDM-T5
12.00
28.25
SELFEVAL
LDM-T5
13.50
29.00
the performance CLIP-based models lower. Thus, as suggested by prior work Yuksekgonul al. (2023), CLIP models exhibit a bag-of-words understanding of the text.
Pixel latent diffusion. We compare the SELFEVAL performance of the pixel diffusion models to that of the latent diffusion models in Figure 5. Among models that use the same text encoder, ie. PDM-T5 and LDM-T5, we observe that the latent diffusion models outperform the pixel diffusion ones in most cases, especially on the harder tasks of Attribute Binding, Count, Spatial Relations and Text Corruption. We hypothesize that this difference can be explained by the fact that the latent diffusion models operate on the compressed latent space and prioritize the text conditioning while 'oflloading' the high-frequency image details the autoencoder. We further investigate the performance of pixel and latent diffusion models by employing human raters to evaluate their text faithfulness in Figure 5. The data, for human evaluation, is constructed by randomly picking 500 examples from the all the tasks (100 examples from each task except text corruption), and choosing the right caption as the text prompt. We convert the accuracy of SELFEVAL to voles, by counting the number of samples where only one model is right. From Figure 5, we observe that human raters prefer the generations of latent diffusion models over pixel diffusion models for text faithfulness. SELFEVAL also shows that latent diffusion models have better text faithfulness and shows an alignment with human evaluations.
Comparison with other metrics. In Tables 2 and 3, we compare SELFEVAL with metrics that utilize external models on PDMs and LDMs, respectively. We compare SELFEVAL with the well- known CLIPScoreHessel et al. (2021) and recently proposed metrics such as MID Kim et al. (2022), LLMScore Lu et al. (2023), and VPEval Cho et al. (2023). In each table, we compute the scores for models using CLIP and T5 text encoders across five tasks (sec Section 4.1) and compare them with human ratings. The row labeled "Human" in both tables indicates the winning votes obtained by each model. All rows use a green or red cell denote whether the ranking among models with CLIP and T5 text encoders agreos or disagrees with human judgment, respeetively. As mentioned in Scction 1, we observe that the value ranges of MID vary wildly, making them uninterpretable and incomparable to the rest. CLIPScore measures the cosine similarity of the generated image with the input text prompt and has range of [], while VPÉval, LLMScore, and SELFEVAL compute accuracy. On the PDMs, we note that each metric disagreed with human judgment on at least one split. On the LDMs, we observe significant. disagreement among different models. CL.PScore and SELFFVAL are the only two models with significantly less disagreement with human raters. In Fig. 6, we show the Spearman's rank correlation p Spearman (1904) between each metric and the human ratings. On the x-axis, we plot P for the PDM's results, and the LDM's results are plotted the y-axis. The shaded region in the plot indicates the desired range for the correlations. Similar to Table 3, we observe that all metrics except EVLFEVAL have a negative correlation with human ratings. SELFEVAL is the only

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Table Comparison of evaluation metrics on pixel diffusion models. compare SELFEVAL with existing metrics on pixel diffusion models. 'Humans' represents the wimning voles obtained by the models in the human evaluation. Each row uses green or red cell to denote whether the ranking agrees or disagrees with human judgment respectively. Note that the value ranges for CLIPScore and MID differ and are not directly comparable to those of the other metrics..
Method
Astribute binding
Color
Count
Shape
Spatial
CLIP
T5
CLIP
T3
CLIP
CLIP
T5
CLIP
T5
Humans
24
117
29
42
41
69
30
7
19
72
CLIPScore (t)
0.90
0.98
0.89
0.91
0.81
0.76
0.86
0.82
0.76
0.78
MID (L)
8.6E14
-2.1E14
-2.7E5
-1.E5
-8.6E3
-1,1E4
-4,6E3
2.6E3
-1,0E15
-5.6E15
VPEval (1)
61.6
63.1
86.9
86.0
16.7
31.9
94.6
91.9
18.1
25.3
LLMScore (1)
6.6
8.2
15.5
15.8
7.3
9.2
10.82
9.9
23.5
25.6
SELFEVAL (1)
50.4
53.3
28.5
30.8
22.8
25.2
33.2
34.3
33.6
34.0
Table Comparison of evaluation metrics on latent diffusion models. We compare SELFEVAL with
existing metrics on latent diffusson models. 'Humann' represents the winning votes obtained by the models in the human evaluation. Each row uses a green or red cell to denote whether the ranking agrees or disagrees with human judgment respectively. We observe significant disagreement with human ratings for metries, except EvAL. and CLIPScore, compared to results on pixel diffusion models.
Methud
Attribute binding
Colur
Count
Shape
Spatial
CLIP
CLIP
CLIP
T5
Humans
14
140
27
69
11
140
25
48
21
73
CLIPScore (1)
0.89
0.99
0.85
0.80
0.76
0.80
0.76
0.86
0.75
0.83
MID (1)
-8.1E14
-1.3E14
-1.1E5
-1.1E5
-2.1E1
-7,6E3
-1.1E3
-2,1E3
4.33E14
-8 8.0E15
VPEval (1)
60.4
66.0
87.5
85.7
64.7
18.9
92.9
92.2
34.1
23.4
LLMScore (1)
9.2
9.1
15.4
17.3
9.6
8.4
11.6
10.6
21.4
23.1 1
SELFEVAL (t)
51.0
54.1
33.0
34.4
24.4
26.3
33.8
33.5
33.4
35.8
metric that correlates well with both PDMs and LDMs. Note that while all existing metrics compute the score generated images, our proposed metric, SELFEVAL, uses the ground truth image-text pair to compute text faithfulness. These findings underscore the robustness of SELFEVAL in aligning with human judgment across different models and tasks, highlighting its potential as a reliable metric.
Qualitative results. Figure (Top) compares the generations pixel diffusion models that use T5 and CLIP text encoders. In each example, the image on the left and right are generated using CLIP and T5 text encoder respectively. We notice that as the difficulty of prompts increases, models with a stronger text encoder performs better. Both the models fail on the much harder task of counting instances and spatial relationships. In Figure 7 (Bottom), each example consists two images generated using pixel diffusion model (left) and a latent diffusion model (right) with a T5 text encoder. We observe that unlike the pixel diffusion model, the latent diffusion model can get small yet important details right ("gray table cloth" and "white handles" in the second and sixth example respectively). We believe that the latent diffusion model can offload the high frequency appearance details to the autocncoder, allowing it pay more attention the conditioning variable.
4.3 Generative models applied to other reasoning tasks
We now use the challenging Winoground Thrush al. (2022) benchmark to evaluate the vasion-language reasoning abilities of diffusion models. Winoground defines two tasks (1) 'text score' that evaluates text retrieval given an image; and (2) 'image score' that evaluates image retrieval given text prompt.
SelfEval vs concurrent work Concurrent work from Li et al. (2023a) demonstrates that diffusion models perform well on the Winoground text score task and achieve competitive performance with

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Table Drawback MID Kim et al. (2022). We compute MID using two CLIP models of two LDM-CLIP models with different backbones. Like CLIP Score, MTD is sensitive to the CLIP model used for evaluation.
0.8
Spearnnn's LDM
0.4
MIDI
Model
0.2
VIT-B/32
ViT-L/14
LDM-CLIP (ViT-L/14)
27.77
25.70
0.0
LDM-CLIP (VIT-H/14)
29.25
23.53
Table 5: Performance of CLIP on the benchmark.
ViEwll
We evaluate the zero-shot performance of CLIP (VIT- L/14) on the six tasks. "Random" is the chance accuracy. CLIP achieves impressive performance on the tasks of
LLMScore
-0.4 SeMEvall
0.0
0.2
0.4
0.6
0.8
Color and Shape. The performance of CLIP is close to random the remaining lasks making it unsuitable for evaluating generative models on such prompts.
Spearman's p for PDM
Figure 6: Spearman's correlation with ground truth across evaluation metrics. We compute the Spearman rank correlation between luman ralings and metrics in Tables 2, 3. Existing metrics show good correlation with pixel diffusion models but a negative correlation with latent diffusion models. In contrast, SELFEVAL positively correlates with both types of models.
Model
Aturibute binding
Color
Count
Shape
Spatial
Text cormuttio
Random
25
25
CLIP
55.40
85.20
67.80
91.10
40.50
51.00
discriminative models. Using their formulation yields poor results (zero accuracy) on the image score task as shown in Table 1. Li et al. (2023a) use the ELBO loss as a proxy for the likelihood p(x c) which works well for comparing different text prompts and thus leads to good text score performance. However, our analysis revealed that the ELBO loss computed for the predictions from two different images are not comparable, and thus leads to zero image score. SELFEVAL on the other hand, doesn't t approximate the likelihood but instead estimates it as described in Sec 3. Using SELFEVAL leads to a non-zero imago-score for the same generative model used by Li et al. (2023a), and yiclds performance close to that of the discriminative CLIP ViT-L model.
SelfEval applied to other diffusson models. Using SELFEVAL reveals that all the diffussin models introduced in Section 4.1.1 achieve competitive performance on both the image score and text score tasks. Compared to all the discriminative CLIP models, generative models achieve strong results in both image and text scores using SELPEVAL. This result reinforces the notion that optimizing the generative objective can provide non-trivial and complementary improvements for several visuo-linguistic reasoning tasks. For additional analysis various hyperparameters on the performance on Winoground, refer to the supplement.
4.4 Drawbacks of existing metrics
In this section, we discuss limitations of metrics computed using external models like CLIP and LLMs that SELFEVAL can effectively address. CLIP score, the most common metric for evaluating text faithfulness of generative models, measures the cosine similarity between the features of the generated image and the conditioned text caption. Recently, Mutual Information Divergence (MID) Kim et al. (2022) used CLIP features to compute the negative Gaussian cross-mutual information between the image and text prompt. LLMScore Lu et al. (2023) evaluates the text faithfulness of text-to-image models by prompting an I.LM to generate a score and rationale given a generated image and a text, prompt. VPEval Cho al. (2023) is an interpretable and explainable metric for text-to-image generation based on visual programming Gupta & Kembhavi (2022)Surís et al. (2023).
Sensitivity to the exact CLIP model. We report the CLIP similarity scores of the generations from two versions of the Latent Diffusson Models Rombach et al. (2022) on prompts from Drawhench Saharia

---

Published in Transactions on Machine Learning Rescarch (12/2024)
F.BE
A blue lighted bridge water night
A bluc and white double decker bus the street
black umbrella the orange dress
gny white door
Five cuhes
in blender
The large pillows and leather headboard
The gray tablocloth and the white
The boulders the right of the house
The the left the tree
An old toilet painted in red with wooden lid
A with white bandles
Figure 7: Qualitative Results. (Top): Each example compares the generations of pixel diffusion models with CLIP (left) and T5 (right) text encoders. As the difficulty of the prompt increases, models with stronger text encoders maintain higher text fidelity. Both the models fail on simple prompts from Count and Spatial relationships. (Bottom): Comparison between generations of Pixel (left) and Latent (right) diffusion models with a T5 text encoder. Latent diffusion models can get smaller details like "gray cloth" and "white handles" (second and last example respectively) correctly.
et al. (2022), Winoground Thrush et al. (2022) and COCO-minival Lin et al. (2014) datasets in Figure 2. The first model (LDM-CLIP (ViT-L/14)) uses the text encoder of CLIP with ViT-L/14 backbone and the second model (LDM-CLIP (ViT-H/14) uses the text encoder with OpenCLIP Ilharco et al. (2021) ViT-H/14 visual backbone. Across all the three datasets, we observe that LDM-CLIP (ViT-I./14) ranks higher than LDM-CLIP (ViT-H/14) if CLIP (ViT-L/14 visual backbone) model is used, but ranks lower with an OpeDCLIP (ViT-H/14 visual backbone). Our hypothesis is that images generaled by a model using a particular CLIP text encoder may still contain some residual information, which could cause them to receive higher scores when assessed using the same CLIP model. This type of bias was identified by Park et al. (2021) in the context of evaluation of text-to-image models, though not in relation to the CLIP score. We observe that this bias is not limited to the CLIP score but pertains to any metric that uses CLIP features.
Table 4 shows the MID Kim et al. (2022) computed using CLIP VIT-B/32 and ViT-L/14 backbones of two models, LDM-CLIP (ViT-I./14) and (ViT-H/14) on prompts from the Winoground Thrush et al. (2022) dataset. Our observations show that when MID is computed using ViT-B/32, LDM-CLIP (VIT-H/14) ranks higher than LDM-CLIP (ViT-L/14). Conversely, when MID is computed using the VIT-L/14 backbone, LDM-CLIP (ViT-L/14) ranks higher than LDM-CLIP (VIT-H/14). We emphasize the need for caution among rescarchers who employ this metric, particularly concerning this bias. SELFEVAL avoids this problem as do not employ an external model for evaluation.
CLIP score is limited by CLIP's performance and thus using it as proxy on tasks where CLIP itself has poor performance does not yield meaningful comparsions. While the CLIP model has demonstrated impressive zero-shot performance on several image-text lasks, it has severe limitations on many complex reasoning tasks. We compute the performance of CLIP ViT-L/14 model on the six tasks introduced in Section 4.1 and report the results in Table CLIP performs well on Color and Shape but its performance on all the other tasks is poor. On the widely used DrawBench prompts, 25% of the captions evaluate the generations for attribute binding, counting, spatial relationships and text corruption. Thus, using CLIP to evaluate generations on such prompts in DrawBench is not ideal. SELFEVAL avoids this problem by directly leveraging the diffusion model itsclf.
Hallucination in LLMs affects evaluation. VPEval Cho et al. (2023) and LLMSScre Lu et al. (2023) employ vision foundation models to ground the concepts, mentioned in the text prompt, in the generated image and leverage a LLM to verify the prompt adherence. This would work if the LLMs are robust, but shown in Fig. 8, LLMs hallucinate irrelevant information making the evaluation unreliable. This is reflected in the worse than chance performance of L.MSccre Lu et al. (2023) in Tables 2, 3 across all the metrics making them unsuitable for such evaluation tasks. We reiterate
12

---

Published in Transactions on Machine Learning Research (12/2024)
VPEval
Captian: siimee
Caption: brows polor gray
Captisn: the skinay borse and the large tree
VPExal Score: 0.2
V PCval Scure:
ViPEva Seore: 0.25
Radionale:
Retienale:
the nchre black? Folos
Thue
black.
Is there poinr bear! False
Rationade:
bour?
The hoose sidnay! False lagge?? the tree large? True
hoss skinay? Faise
The mone nrimmed. Fable ciimeed Falae
The polar bear brown, Frise grny. Palse
LLMScore
Captien: dog sleaping
Caption: grocs
Caption:
elephort
young
LLMiscorr: 92
LLMSScre:
LL
Rational::
Rationale::
Rationale:
image closely aligns with the text prompe, depiccing paseefully resting cozy couch with pillow bencath
image description
capoures
perfectly align with the depicted.
the poompe majestic
elephanr front young bush. The image aligns perfeetly prompe.
textines, giraffe and tree sonumnoly portnyyd
Figure 8: Hallucination in large language and vision models affects evaluation of generative models. (Top): Three examples showing the caption, entries are highlighted in red. We observe that VPEval penalizes the generation despite its faithfulness to the text, due halluciiations of the vision and language model. (Bottom): Three examples showing the generated image, a wrong caption and the rationale generated by LLMScore Lu et al. (2023). LLMScore assigns high score despite the incorrect caption, due the hallucinations of the LLM, as highlighted in red the rationale.
and corresponding rationale from VPEval Cho et al. (2023). Wrong
that the performance of the external model on the evaluation set has a large impact on its evaluation capabilities and SELFEVAL is such way to eliminate the reliance external models for evaluation.
5 Conclusion and Limitations
This paper introduced SELFEVAL, an automated method for evaluating the text-understanding capabil- ities of diffusion models. SELFEVAL estimates the likelihood of real images given text prompts using the diffusion model itself, eliminating the need for external discriminative models. Our experiments demonstrated that SELFÉVAL aligns with human evaluations across various models and tasks, proving its reliability metrics will expedite diffusion model research and encourage further improvements. SELFEVAL's applicability extends beyond text-to-image diffusion models, potentially serving in the evaluation of other conditioned diffusion models like text-lo-audio and texl-to-video. However, in its current form, SELFEVAL is designed to work with generative modeling methods that estimate likelihood, and while there are workarounds, they are not trivial. Additionally, we assume that the generative model has been trained sufficiently, as the overall SELFEVAL score inherently depends on the training quality of the generative model itself. Future work aims to generalize SELF EVAL for with non-diffusion--aase generative models.
automated metric for text-conditioned image generation. We anticipate that such
6 Broader Impact
This research introduces SELFEVAL, an automatic evaluation method, eliminates the use of an external model for evaluation while demonstrating strong agreement with human annotators. This automated evaluation could significantly reduce the monetary and time costs associated with hiring human evaluators, making the process of developing and refining generative models more elficient. Progress in research areas related to SELFEVAL could potentially improve the ability of generative models to adhere to text. However, such models pose serious concern as they could lead to the creation of deceptive images if placed in the wrong hands.

---

URL http://21..mmmmmmeeeeme Published in Transactions on Machine Learning Rescarch (12/2024)
References
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 3
Jaemin Cho, Abhay Zala, and Mohit Bansal. Visual programming for text-lo-image generation and evaluation, In NewrIPS, 2023. 9, 11, 12, 13
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero-shot classifiers. In ICLR 2023 Workshop on Multimodol Representation Learning: Perks and Pitfolls, 2023. URL https:
4
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenct: A large-scale hierarchical image database. In 2009 IEEE Conference on Compuler Vision and Pollern Recognition, pp. 248 255, 2009. doi: 10.1109/ 20000//VVR.2000.22017 5206844 21
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. ArXiv, abs boss/21050000 0.023,, 2021. URL https://api. semantccscholar org http/oorpuiD::2333300
3
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63:139 144, 2014. URL https://api semanticscholar. wwwpeholar.rrrr//eeeeeeooeeeeeee CorpusID :10.pppSSD::220900.. 12209503.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming:
Compositional visual reasoning without training. 2023 IEEE/C' /CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14953-14962, 2022. URL https: /tappp..smaaiiscccolllaaae Drraaa/ai :t 253734854.
Xuehai He, Weixi Feng, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Discriminative diffusion models as few-shot vision and language learners, 2023. 4
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference- free ovaluation metric for image captioning. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Domínican Republic, November 2021. Association for Computational Linguistics. doi: https://aclanthol/
org/2021. emnlp caamlppmain... 595.....95.... 9
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria- Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Newral Information Processing Systems 2020, NewrIPS 2020, December 6- 6-12, 2020, virtual, 2020. URL https://proceedings. neurips.... cctpcccpp///pcccccccccc /2000/ haao//aash/a/aee
Abstract. 2, 3
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. orXiv preprint ar 230: 1 11897, 2023. 7, 21
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL https://doi.org: wtw.://022..2..5555552 you use this software, please cite it as below. 8, 12
If
Justin Johnson, Bharath Haríharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 1988 1997, 2016. URL httpss//pp...mmatiicccaaaaa : 15458100. 7
semanticecholar

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Jin-Hwa Kim, Yunji Kim, Jiyoung Lee, Kang Min Yoo, and Sang- Woo Lee. Mutual Information Divergence: A unified metric for multimodal generative models. In Advances in Neural Information Processing Systems 35, 2022. URL https://oppnrrever. peot/forumidd-wkGee.
2, 9, 11, 12
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Truck Proceedings, 2014. URL http://arxiv, org/ abs//312. 6114. 2
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision, 123(1):32 73, may 2017. ISSN 0920-5691. doi: 10.1007/s112233-06-09853 https://doi. org/ 10. wtt.:1000011110001001 016-091--7 7
URL
Martha Lewis, Nihal V. Nayak, Qinan Yu, Jack Merullo, and Elizabeth-Jane Pavlick. Does clip bind concepts? probing compositionality in large image models. ArXiv, abs/2212.10537, 2022. URL
https://api. semanticscholart wtwp:/orrrrrrr/rorrtrrr..ra
:5:28877765 7
Alexander Cong Li, Mihir Prabhudesai, Shivam Duggal, Ellis Langham Brown, and Deepak Pathak. Your diffusion model is secretly a zero-shot classifier. In JCML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling, 2023a. URL https://openreviil hwtp:/.ee//teeeeeeeeee net/forumidd:433yKaa 4, 9, 10, 11,21 21
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22511 22521, 2023b. URL https://api. swmantcccchalaanr. wwwpaacchooaaaaoorrrrruuuuu
255942522. 3, 20
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1. 7, 12
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey 'Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. ICCV, 2023. 3
Yujie Lu, Xianjun Yang, Xiujum Li, Xin Eric Wang, and William Yang Wang. Llmscore: Unveiling the power of large language models in text-to-image synthesis evaluation, 2023. 9, 11, 12, 13
Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swami- nathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava, Diffusion models beat gans on image classification, 2023.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, 1lya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with gextdgd diffusion models. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song. Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Marylend, USA, volume 162 of Proceedings of Machine Learning Research, pp. 16784 16804. PMLR, 2022. URL https: mlr, /rres/vv62// nicho122a. html. 3
Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to-image synthesis. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: http::////peeeeiie. .net/ forum? id=bKEhQhPeKaF. 4, 12

---

Published in Transactions on Machine Learning Rescarch (12/2024)
William Peebles and Saining Xie. Scalable diffusion models with transformers. 1CCV, 2023. 3
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. CoRR, abs/1511.06444, 2015. URL attps://apt
semanticscholar wrw.aaaursrrCooorI...... 11T.::61..... 2
Alce Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2,
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text- to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
html. 3,
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. 3, 8
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022. 3, 8, 11
Chitwan Saharia, William Chan, Saurabh Saxena, Tala Lá, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to- image diffusion models with deep language understanding. ArXie, abs/2205.1144 11487, 2022. URL https://apls www.eeeeeaaaanniieeeeeeee org/ Ctse/Crrpussl ID: 248986576. 3, 7, 8, 11
Jascha Sohl-Dickstein, Fric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Lille, France, 07-09 Jul 2015. PMLR. URL https::/
Learning Research, pp. mlr. mlr.prestee Har.press/v37/ 1 htte//shllddssshse wiiiddiiisssiinni html. 3, 4
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021. URL https://openreviin wtt.eviwwwwee/oommmi..iii.
StigiarCHLP. 8
C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904. ISSN 00029556. URL http://www.jstor httg//taalee1111111 9
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python exccution for reasoning. Proceedings of IEEE International Conference on Compuler Vision (ICCV), 2023. 11
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality: 2022 LEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), pp. 5228-58 5238, 2022. URL attps://api. httpps///pppssmmnntcccccooaaa wwt.sschhhoaaoooooropaassseeeess org/CorpusID 248006414. 4, 10, 12, 20
semanticscholar
Ziwei Xu, Sanjay Jain, and Mohan S. Kankanhalli. Hallucination is inevitable: An innate limitation of large language models. ArXiv, abs/2401.11817, 2024. URL attps://app semanticacholaat org/
CorpusID: CorpusID:26706921 267069277. 2
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision- languu- models behave like logsoof-oords, and what to do about it? In International Confer- ence on Learning Representations, 2023. URL https : httpp//arrrreeeeetee/trrraaaaaaa wwt.:/orrmm..--ALLvvaaa
7, 9
16

---

Published in Transactions on Machine Learning Rescarch (12/2024)
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023. 3
17


Reviews:
[{"review_id": "TJNqvk7tYp", "reviewer": "Reviewer_fHRR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. A major plus is that SelfEval doesn’t rely on any external models, so it avoids the dependencies and potential biases that external tools might introduce.\n2. The authors have run extensive experiments, and the results show that SelfEval’s evaluations are closely aligned with human judgments across various tasks. This really boosts its reliability as a tool for model assessment.\n3. SelfEval digs deep, evaluating specific tasks like color recognition, spatial relationships, and counting. This approach provides detailed insights into how well a model understands the visual-text alignment, which is super valuable.\n4. Another strong point is that SelfEval is flexible, it’s built to work with any generative model that can calculate likelihoods, so it’s broadly applicable across different text-to-image models.\n\n### Cons\n1. One potential limitation here is that if the generative model itself isn’t well-trained or is being applied to unfamiliar tasks, its accuracy and reliability in SelfEval could suffer. This means the scores might not truly reflect the model’s performance on such tasks, which could impact the reliability of the evaluation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SelfEval, a method to evaluate text-to-image generative models by converting them into discriminative models. SelfEval assesses a model's text-faithfulness and visual understanding without relying on external models like CLIP, which the authors argue introduces biases and limitations. The method computes the likelihood of real images given text prompts and evaluates the model’s performance across tasks like color recognition, counting, and spatial understanding. SelfEval demonstrates a high correlation with human evaluations, potentially offering a scalable and objective evaluation tool for generative models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. How well does SelfEval handle more complex spatial relationships or compositions? It would be helpful to see some analysis on this front to understand how it performs in scenarios where there’s high visual-text alignment complexity.\n\n2. Could SelfEval be adapted to assess image quality in addition to text-faithfulness? Right now, the method focuses only on text-faithfulness, so it would be interesting to know if it could also be modified to evaluate overall image quality.\n\n3. Since SelfEval’s approach works best with models that compute likelihoods directly (like diffusion models), does this limit its applicability to other generative models, such as GANs, that don’t have a straightforward likelihood-based mechanism?", "broader_impact_concerns": "The broader impact has been discussed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lRnx1Ak7tP", "reviewer": "Reviewer_WmNq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Not having to rely on external models for evaluation is nice. Also automated evaluation as opposed to requiring human intervention. This gives us a lot of flexibility. \n- Experiments show the benefits of SelfEval, including attribute binding, color and shape recognition, spatial relations, and text corruption.\n- Results show good consistency between SelfEval and human evaluation.\n\n**Weaknesses**\n\nThe paper is solid and does a thorough job of evaluating SelfEval.  I did not find any significant weaknesses.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed SeflEval, an approach to evaluate the text-to-image alignment of generative models. The key idea is to use the generative model itself to perform this evaluation. It does this by computing the likelihood of the text prompt on real images rather than generated images. Extensive experiments were conducted. Results showed that SelfEval shows more text faithfulness than using CLIP scores and also more aligned with human evaluation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can you comment on how well the likelihood scores are calibrated across different generative models? This is critical for comparing different generative models (e.g., pixel vs. latent models, as shown in the paper) with each other. Is there a way to verify if the score is well-calibrated or not?", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sfNfvwHRHD", "reviewer": "Reviewer_H2un", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Clarity: The \"Preliminary\" and \"Related works\" sections are written in sufficient detail that even non-experts can understand them. I believe this provides a good explanation for beginners in the field. However, the explanation of how \"SelfEval\" actually works is severely lacking. If I understand correctly, the only structural explanation of SelfEval is in Section 3.3 on page 6, and even this does not make it clear how SelfEval intends to perform evaluations.\n\nNovelty and Soundness: As I understand it, this paper ultimately proposes using the reverse process of a generative model to estimate \n$p(x∣c)$, and then using the highest likelihood to perform classification. Then it measures the 'matching performance' of each generative model. While the abstract is written in a grandiose manner, the summary in Section 3.3 suggests that this SelfEval works only with finite number of captions. As a reviewer who is not highly experienced in experiments in this field, I would appreciate a clear summary before the experiment explanation, particularly regarding what happens if the image is not part of the set of captions, or whether the classification is only for images within the set of captions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present an automated approach for evaluating the text alignment of text-to-image generative diffusion models using standard image-text recognition datasets. Their method, referred to as SelfEval, utilizes the generative model to compute the likelihood of real images given text prompts, which can then be employed to perform recognition tasks with the generative model. \n\nThe authors evaluate the generative models on standard datasets designed for multimodal text-image discriminative learning, assessing fine-grained aspects of their performance, including attribute binding, color recognition, counting, shape recognition, and spatial understanding. \n\nUnlike existing automated metrics that rely on an external pretrained models which are sensitive to the specific pretrained model and its limitations, SelfEval circumvents these issues. If the authors' claims are true, this is the first automated metric that demonstrates a high degree of agreement with gold-standard human evaluations for measuring text-faithfulness across multiple generative models, benchmarks, and evaluation metrics. \n\nAdditionally, SelfEval reveals that generative models exhibit competitive recognition performance on challenging tasks such as Winoground image-score when compared to discriminative models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1. The authors should explain with more details about how SelfEval works. Is it only works for the fixed classes (captions)? What is the prior distribution over the classes? If so, what is the prior distribution over the captions? Uniform? If their result is not limited to the classification over fixed captions, how one could define the prior distribution? How did you generate and select distractors in Figure 3? I definitely feel that they should explain their general framework, much more than the things in Section 3.3.\n\n2. Also, the authors should also explain how to measure the closeness of the captions generally, before the start of the experiment section. For example, suppose that in Figure 3, Red sphere is quite close to Red cylinder, but not close to blue tiger (for example). Do you measure the closeness between two captions? Or regarding it as 0-1 (1 if correct, 0 otherwise)? If the details change over tasks, then the author should clarify what is 'common structure' of SelfEval before the experiment, and what can be fine-tuned.", "broader_impact_concerns": "No concerns yet.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The idea of the paper is simple, yet practical. This idea could be potentially applied to other alignment problems besides text-to-image. It allows evaluating text-to-image models without relying on external pre-trained models, whose biases and limitations could affect the accuracy of the evaluation. The authors support their proposed method, SelfEval, using a large set of experiments.    

There are a few issues raised by the reviewers that are important to be addressed. I do no think it is needed to accept the paper with minor revision, but I strongly encourage the authors to try their best to address them.

1) Dedicate a paragraph or a section to list/highlight the limitations of SelfEval, including (i) it works with models that directly compute likelihood, e.g., diffusion models; (ii) if the generative model is not well-trained, its evaluation could be questionable; (iii) its success depends on how well the discriminative task is designed/selected (e.g., making sure to have hard negatives ...).

2) Better and more clearly explain how SelfEval works. This is the main contribution of the paper and the section dedicated to it (Section 3.3) is not well-written/self-contained. It is important to explain how the likelihood is computed. You briefly explain it in both Sections 3.2 and 3.3, why? Why not just explain it in one place and clearly mention how the likelihood is computed. It is also important to clarify how the discriminative task is selected and how the evaluation metric is computed. Perhaps a simple example of a discriminative task can help.
Decision: Accept (poster)