ImagenHub: Standardizing the evaluation of conditional image generation models
Authors: Max Ku
Tianle Li
Kai Zhang
Yujie Lu
Xingyu Fu
Wenwen Zhuang
Wenhu Chen
Venue: ICLR 2024 poster (2024)
Keywords: Image Generation
Image Editing
Evaluation
Benchmark
Diffusion Model
Primary area: datasets and benchmarks
TL;DR: We present ImagenHub, which is a continuous effort to standardize the inference and evaluation of all the existing conditional image generation models. We present lots of key takeaways in the paper to help develop better image generation models.

Abstract: Recently, a myriad of conditional image generation and editing models have been developed to serve different downstream tasks, including text-to-image generation, text-guided image editing, subject-driven image generation, control-guided image generation, etc. However, we observe huge inconsistencies in experimental conditions: datasets, inference, and evaluation metrics -- render fair comparisons difficult.    
This paper proposes ImagenHub, which is a one-stop library to standardize the inference and evaluation of all the conditional image generation models. Firstly, we define seven prominent tasks and curate high-quality evaluation datasets for them. Secondly, we built a unified inference pipeline to ensure fair comparison. Thirdly, we design two human evaluation scores, i.e. Semantic Consistency and Perceptual Quality, along with comprehensive guidelines to evaluate generated images. We train expert raters to evaluate the model outputs based on the proposed metrics. Our human evaluation achieves a high inter-worker agreement of Krippendorff’s alpha on 76\% models with a value higher than 0.4. We comprehensively evaluated a total of around 30 models and observed three key takeaways: (1) the existing models’ performance is generally unsatisfying except for Text-guided Image Generation and Subject-driven Image Generation, with 74\% models achieving an overall score lower than 0.5. (2) we examined the claims from published papers and found 83\% of them hold with a few exceptions. (3) None of the existing automatic metrics has a Spearman's correlation higher than 0.2 except subject-driven image generation. Moving forward, we will continue our efforts to evaluate newly published models and update our leaderboard to keep track of the progress in conditional image generation.

Published as a conference paper at ICLR 2024
IMAGENHUB: STANDARDIZING THE EVALUATION OF CONDITIONAL IMAGE GENERATION MODELS
Max Ku Tianle Li Kai Zhang', Yujie Lu Xingyu Fu Wenwen Zhuang Wenhu Chen University of Waterloo Ohio State Universityf, University of California, Santa Barbara University of Pensylvania Central South University
(max. ku, t291i, wenhuchen/@uwatter.o..
https://tiger-ai--ae githuu io/ ImagenHub/
ABSTRACT
Recently, myriad of conditional image generation and editing models have been developed to serve different downstream tasks, including text-to-image genera- tion, text-guided image editing, subject-driven image generation, control-guided image generation, etc. However, we observe huge inconsistencies in experimen- tal conditions: datasets, inference, and evaluation metrics render fair compar- isons difficult. This paper proposes ImagenHub, which is one-stop library to standardize the inference and evaluation of all the conditional image generation models. Firstly, we define seven prominent tasks and curate high-quality evalua- tion datasets for them. Secondly, we built a unified inference pipeline to ensure fair comparison. Thirdly, we design two human evaluation scores, i.e. Seman- tic Consistency and Perceptual Quality, along with comprchensive guidelines to evaluate generated images. We train expert raters to evaluate the model outputs based on the proposed metrics. Our human evaluation achieves a high inter-worker agreement of Krippendorff's alpha on 76% models with value higher than 0.4. We comprehensively evaluated a total of around 30 models and observed three key takeaways: (1) the existing models' performance is generally unsatisfying except for Text-guided Image Generation and Subject-driven Image Generation, with 74% models achieving an overall score lower than 0.5. (2) we examined the claims from published papers and found 83% of them hold with a few exceptions. (3) None of the existing automatic metrics has Spearman's correlation higher than 0.2 except subject-driven image generation. Moving forward, we will con- tinue our efforts to evaluate newly published models and update our leaderboard to keep track of the progress in conditional image generation.
INTRODUCTION
ImagenHub
ImagenHub Inference Library
ImagenHub Evaluator
Text2lmoge vegstabiss
topenealism truis
Semantic Consistency
import imagen_
Control-guided
(SC)
model = imagen hub.load(name)
ImageEditing
img = mooell.gmeeateecooddeeee
Perceptual
API
Inference
Raters
Quality (PQ)
Figure 1: The overview of ImagenHub framework, which consists of the newly curated ImagenHub dataset, ImagenHub library, and ImagenHub evaluation platform and standard.
With the recent development of diffusion models, image generation has quickly become one of the most popular research areas in AI. To enable controllability in the image generation process, a myriad of conditional image generation models have been proposed in the past year. Diverse set of conditions have been attempted to steer the diffusion process. One of the most popular tasks is text-guided image generation (Ramesh et al., 2022; Rombach et al., 2022; Saharia et al., 2022),

---

Published as a conference paper at ICLR 2024
which aims to ground on text prompt generate the corresponding image. Besides that, there also subject-conditions image generation (Gal et al., 2022; Ruiz et al., 2023), text-guided image editing (Brooks et al., 2023), cuulii-subecc--coddiionee image generation (Kumari et al., 2023), style-guided image generation (Sohn et al., 2023), etc. These different tasks aim to serve different types of downstream applications by enabling subject-level, hackground-level, style-level controls. The field is evolving at an unprecedented pace and lots of improvements have been reported in the published papers. However, one glaring issue we observed is the published work are highly inconsistent in their experimental setups. To summarize, the inconsistencies mainly come from three aspects, namely dataset, inference and evaluation:
Dataset: The existing work curated their own evaluation dataset, which makes the comparison of different models totally incomparable.
Some work requires heavy hyper-parameter tuning and prompt engineering to achieve reasonable performance, which makes the model less robust. Due the tuning effort on different models differe significantly, their comaprison could become unfair.
Inference:
The existing work used different human evaluation protocols and guidelines. This inconsistency renders it impossible to compare human evaluation scores across different meth- ods. Moreover, some the work either employs a single rater or does not report inter-worker agreement. Thus, the reported results might not be comparable across different papers.
Evaluation:
0.8
DallE-3
Overall Score of different subtasks
Best
Avg
0.6
MagicBrush
0.4
SDXL-Inpaint
CustomDiff
PhotoSwap
ControlNet
0.2
Text-guided
Subject- Imagen
Text-guided ImgEdit
Mask-guided ImgEdit
Subject-Driven ImgEdit
Multi-concept ImComp
Control-guided Imagen
Imagen
Figure 2: The best and the average model performance in each task
These three inconsistencies make it nearly impossible to track the real progress in the field of the conditional image generation. Such an issue could greatly hinder the development of this field. The desiderata is to build centralized effort to fairly evaluate every model. More importantly, this effort needs to be continuous to keep up with the evolvement of the field. Our paper aims serve this purpose to standardize the serving and evaluation of all open-source conditional image generation models. More specifically, ImagenHub consists of the modules listed Figure 1.
We surveyed the existing public evaluation sets for all the generation tasks and then picked diverse instances from them to build our ImagenHub dataset. ImagenHub dataset consists of 7 subsets, each with 100-200 instances. This dataset aims standardizing the evaluation input to ensure fair comparison for different models.
ImagenHub Dataset.
ImagenHub Inference Library. We built ImagenHub library' to evaluate all the conditional image generation models. We ported the highly dispersed codebase from the existing works and then standardized them into unified format. During inference, we fixed the hyper-parameters and the prompt format to prevent per-instance prompt or hyper-parameter tuning, which makes the inference of different models fair and reproducible. The library is designed be easily extendable. Our Appendix A.7, A.8, and A.9 show how a third party and researchers can benefit from our work.
ImagenHub Evaluator. We explored different human evaluation metrics and iterated over different versions of the rating guidelines to improve the inter-rater agreement. We settled on two three- valued rating metrics 'Semantic Consistency' and 'Perceptual Quality' to achieve generally high wnter- agreement measured by Fleiss Kappa (Fleiss & Cohen, 1973) and Krippendorff's Alpha (Krippendorff, 2011). We designed rating standard achieve several benefits: (1) Our rating guide is an unambiguous checklist table such that the rater can rate the image with ease. (2) The
it's similar Huggingface libraries (Wolf et al., 2019; Platen al., 2022)
2

---

Published as a conference paper at ICLR 2024
designed rating guideline is unified on every task type. (3) Sustainability. Since each model is rated individually, previous evaluation results can be reused when new models are added.
We demonstrate our evaluation results in Figure 2, where show the overall score of the best- performing model and the medium- medium-performin performing model. Based on our evaluation results in section 5, made some general observations:
The existing models' performance is generally unsatisfying except for Text-guided Image Gen- eration and Subject-driven Image Generation.
We found that evaluation results from the published papers are generally consistent with evaluation, 83% of the published result ranking is consistent with our ranking.
Automatic evaluation metrics do not correlate well with human preference well except subject- driven image generation. The correlation scores are lower than 0.2.
2 THE PROBLEM OF CONDITIONAL IMAGE GENERATION
The goal of conditional image generation is to predict an RGB image y E R3xHxW 3 where H and W are the height and width the image. The prediction of the target image is given a set of input conditions X [et, .2.-..), where X E X. where C₁ denotes the i-th condition. In the problem, we aim at learning prediction function f : X + y with deep learning models. Here mainly consider f parameterized with diffusion models. Here we list a set of tasks consider in Figure 3, where e can be represented as text prompt, image mask, subject image, source image, background image, control signal, etc.
Task Definition. We formally define the tasks we consider as follows:
Text-guided Image Generation:
f(p), where p is text prompt describing scene. The goal is to generate image consistent with the text description.
Mask-guided Image Editing: y = [p, Imusk Iue). where Imust a binarized masked image, and I source image. The goal is to modify given Iwe in the masked region according to p.
Text-guided Image Editing: y f(p L. It's similar to Mask-guided Image Editing except that there is no mask being provided, the model needs figure out the region automatically.
Subject-driven Image Generation: which normally ranges from 3-5. The goal to generate an image according to p regarding the subject S.
fSS p), where S is set of images regarding a specific subject,
Subject-driven Image Editing: y = f(S, p, Im)), where Isc is a source image and S the subject reference. The goal is to replace the subject in I.. with the given subject S.
Multi-concept Image Composition: y fSS, S2, p, Tue). where Si and Sa are
of concept images. The goal to compose them together to generate a new image according to text description p.
Control-guided Image Generation: canny edge, bounding box, etc. The goal is to generate an image following the low-level visual cues.
f [Joount p). where Icomal is the control signal like depth map,
RELATED WORK
Multimodal Conditional Image Synthesis. Recent works in multimodal conditional image synthe- sis often rely on pre-trained vision-language models such as CLIP (Radford et al., 2021) and pre- trained large diffusion models like Stable Diffusion (Rombach al., 2022). CLIP bridges the gap between textual descriptions and visual content, while Stable Diffusion is a latent diffusion model trained on a massive dataset. CLIP and Stable Diffusion are now widely used as core components in applications such as text-to- -mage generation, text-guided image editing (Patashnik et al., 2021; Meng et al., 2021; Couairon al., 2022; Mokady et al., 2022; Zhang et al., 2023), and mask mask-guided image editing (Avrahami et al., 2022; Nichol et al., 2022). Another interesting research direction is image subject personalization with a few or even one image of the subject. Textual Inversion (Gal et al., 2022) optimizes a text token vector represent the subject, while DreamBooth (Ruiz et al., 2023) finetunes the Stable Diffusion model to learn the new subject concept efficiently. These works foster research in potential applications like subject-driven image generation and editing (Li et al., 2023a;b; Gu et al., 2023), and even multi-concept image compositions (Kumari et al., 2023). To explore more control conditions, ControlNet (Zhang & Agrawala, 2023) proposed the usage of zero
3

---

Published as a conference paper at ICLR 2024
C1
C2
C2
Task
y
cartoon styled alarm clock
Text-to-lmage
Generation
Change frisbee to football
Mask-guided Image Editing
Make it a slice of pizza instead of the sandwich
Text-guided Image Editing
[V] dog in
Subject-Drive Driven Image Generation
Versailles hall mirrors
Replace glasses with [V] glasses
Subject-Driven
Image Editing
cat [V] standing by
Multi-Concept
Image Composition
pot [M]
A small dog is curled up top shoes
Control Image Generation
Figure 3: The visualization of all the conditional image generation tasks. Here we consider tasks with 1-3 conditions, where means empty. The special token [V] and [M] are special identifiers.
convolution the Stable Diffusion model to support additional guided image control. This work brings up the idea of the control-guided image generation task and inspired later work (Qin et al., 2023) improving the control versatility.
AT-generated Image Assessment. Evaluating Al-generated images holistically is a complex and open problem (Salimans al., 2016). Researchers have proposed various automatic metrics. In the image quality aspect, Inception score (Salimans et al., 2016), FID (Heusel et al., 2017) are often used. These methods rely on statistics from InceptionNet pre-trained the ImageNet dataset. Despite being widely adopted due to their sensitivity to small changes in images, these metrics are not ideal. They are biased towards the ImageNet dataset, resulting in inadequate evaluations (Borji, 2021). Later works like LPIPS (Zhang et al., 2018) and DreamSim (Fu et al., 2023) proposed better ways to measure the perceptual similarity. In the semantic consistency aspect, the CLIP score (Hessel et al., 2021) is often used to measure the vision-language alignment between the generated image and the prompt. Researchers also worked on alternative methods such as BLIP score (Li al., 2022) and RewgeReward (Xu et al., 2023). However, in tasks like subject-drivg generation and editing, the automatic measurement of semantic consistency is still an open problem. One long-established yet effective approach to assessing Al- Al-generated image performance is to rely on human annotators to assess the visual quality (Denton et al., 2015; Isola et al., 2017; Meng al., 2021; Chen et al., 2023). The downside is that it entails reliance human judgment, which can introduce subjectivity and potentially limit scalability. To mitigate the downsides, the human evaluation design has to be unambiguous and easy follow.
image
METHOD
4.1 HUMAN EVALUATION METRICS
Our proposed metric can be used all seven tasks with the same standard. We adopt two major evaluation metrics, namely semantic consistency SC' and perceptive quality PQ. These two metrics measure the quality of the generated images from two aspects. The semantic consistency measures how well the generated image is aligned with the condition X (ci, .... Specifically, we define
4

---

Published as a conference paper at ICLR 2024
Condition I
Condition
Condition 3
SC rating
Inconsistent
Any
Any
0
Any
Inconsistent
Any
Any
Any
Inconsistent
Partially Consistent
Any
Any
0,5
Any
Partially Consistent
Any
0.5
Any
Any
Partially Consistent
0.5
Mostly Consistent
Mostly Consistent
Mostly Consistent
1.0
Subjects in image
Artifacts
Unusual sense
PQ rating
Unrecognizable
Any
Any
0
Any
Serious
Any
0
Recognizable
Moderate
Any
0.5
Recognizable Any
Moderate
0.5
Recognizable
Little/None
Little/None
1.0
Table 1: Rating guideline for computing the SC and PQ score. Detail in subsection A.2.
1.0
0.9
0.8
Perceptual Quality (PO) Average
0.7
0.6
0.5
0.4
Subject-driven Image Editing
0.3
Text-guided Image Generation
Control-guided Image Generation
0.2
Mask-guided Image Editing
Text gexted Image Editing
0.1
Subject-driven Image Generation
Multi- colti-concept Image Composition
0.0
1.0
Semantic Consistency (SC) Average
Figure Model performance and standard deviation in each task.
the semantic consistency score
SC'y, X)
(1)
where is a modularized function to compute the consistency between y and single condition We set (1) [0, 0.5, 1). where means inconsistent, 0.5 means partially consistent and means fully consistent. With this formulation, as long as the output is inconsistent with any of the conditions, the evaluation score should become zero. Otherwise, the aggregation function will pick the lowest consistency score from all the conditions. On the other hand, perceptive quality measures the image quality, i.e. whether the image contains artifacts, is blurry, or has an unnatural sense. set perceptive quality PQ [0, 0.5, 1], where 0 means extremely poor quality, 0.5 means the image has an acceptable quality and means high quality. In these experiments, each model is rated individually. We train human raters to estimate the g function and PQ function with comprehensive guidelines in Table We derive O /SC PQ as the overall rating of a model. One benefit of using geometric mean the design choice that the rating is penalized when one of the aspect scores is too low. We further studied the configuration in section 5.
5

---

Published as a conference paper at ICLR 2024
Task
Duta Source
Inference Dataset size
DrawBench (Saharia al,, 2022)
DiffusionDB (Wang et al. 2022)
Text-guided Image Generation
197
ABC-6K (Feng et al.,, 2023)
Ours
MagicBrush (Zhang al., 2023)
Mask-guided Image Editing
179
Ours
MagicBrush (Zhung al., 2023)
Text- Text-guided Image Editing
179
Ours
SuTI (Chen et ct. 2023)
Subject-driven Image Generation
150
Ours
CustomDDiflusion (Kumari et al,, 2023)
Multi- Multi-concept Image Composition
102
Ours
Subject-driven Image Editing
DreamEddilemcc (Li et al,, 2023b)
154
Control
Control-guided guided Image Generation
HuggingFace community
1.50
Table 2: All the human evaluation datasets from seven core tasks.
4.2 DATASET AND AVAILABLE MODELS
We present standardized dataset for each type of task. The information of the datasets is shown in Table Some models have different standards of imputs in one task. For example, in the text-guided image editing task, DiffEdit is global description-guides while InstructPix2Pix is instruction-guiie We manually created the equivalent meaning prompts for both methods so they can be aligned. All datasets contain huge variety of test cases mimic the diversity in real-life situations. We hosted all our datasets on the HuggingFace dataset for easy access and maintenance. Here we demonstrate all the evaluated models in Table 3.
EXPERIMENTAL RESULTS
Experiment Setup. All the models either used the default setting from the official implementation or the setting suggested in HuggingFace documentation (von Platen et al., 2022). We disabled negative prompts and any prompt engineering tricks to ensure fair comparison. We conducted human evaluation by recruiting participants from Prolific to rate the images, and our own researchers also took part in the image rating process. We assigned 3 raters for each model and computed the SC score, PQ score, and Overall human score. Then we computed the Fleiss kappa (Fleiss & Cohen, 1973) for each mentioned score. We also computed Krippendorff's Alpha (Krippendorff, 201 I), which is expected to yield a higher value than Fleiss kappa. This distinction arises from the nature of the rating categories, with Fleiss' Kappa assuming nominal categories and Krippendorff's Alpha accommodating ordinal inputs. Both Fleiss' Kappa and Krippendorff's Alpha are bounded within the range of [-1, 1]. where the value > 0 indicates an agreement and closer proximity to indicates a higher degree of agreement. The design choices are explained in subsection 5.2 and Appendix A.6.
Results. In Figure 4, we present an overview of the model performance across various tasks. Our findings indicate that the performance of the current models is generally underwhelming, with the exception being Text-guided Image Generation and Subject-driven Image Generation, which have models reaching higher than 0.6 both SC and PQ averages. The detailed report each model's performance is shown on Table 4. We noticed that the overall automated metrics' correlation with the SC score and PQ score in each task below 0.2 except subject-driven image editing task. Metric values are at Table 5 and Table 6.
5.1 DISCOVERY AND INSIGHTS
We observe that all models are able to generate high-quality im- ages. Regarding semantic consistency, all models have good understanding of the general prompts, while Stable Diffusion XL is better at understanding complex prompts. For example, it exhibits a high degree of accuracy and detail the prompt "A panda making latte art." while other models
Text-Guided Image Generation.

---

Published as a conference paper at ICLR 2024
Model
#Params
Runtime
Keywords for technical detail
Im-ge Generation
Dalle-3 (openai, 2023)
Recaptioned training data
MidJournyy (midjourney, 2023)
Dalle-2 (Ramesh et al., 2022)
3,5B
10s
unCLIP, two-stage
Stable Diffusion (Rombach et al., 2022)
0.8B
38
Latent Diffusion
DeepFloyddF (deep floyd.ai, 2023)
4,3B
37s
Cascaded Pixel Diffusion
OpenJourney (openjourney.ai, 2023)
0.8B
3s
SD, Midjourney data
Stable Diffusion XL (stability.ai, 2023)
2.3B
1ls
Stable Diffusion, X-Large
Mask-guided Image Editing
BlendedDiffusion (Avrahami et al., 2022)
0.5B
57s
Noise Blending, DDPM+CLIP
GLIDE (Nichol et al., 2022)
3.5B
19s
CLIP, Diffusion
SD- SD-Inpaint (runwayml, 2023)
1.1B
1ls
SD, Inpainting training
IDXL-Inpaint (stability.ai, 2023)
2.7B
368
SDXL, Inpainting training
Text- Text-guided Image Editing
SDEdit (Meng et al,, 2021)
1.3B
13s
SDE Prior
Text2Live (Bar-Tal al., 2022)
3.1M
36s
Zero-shot, Edit layer
DiffEdit (Couairon al., 2022)
1.3B
29s
Mask estimation
Cycle Diffuusion (Wu Torre, 2023)
1.1B
9s
DPM-Encoder, Zero-shot Cross-Attention
Prompt-to-Prompe (Mokady et al., 2022)
1.1B
2m
Pix2PixZero (Parmar et al., 2023)
1.1B
21s
Cross-Atiention, Zero Prompt
InstructPix2Pix (Brooks et al., 2023)
1.1B
11s
SD, synthetic P2P data
MagicBrush (Zhang et al., 2023)
1.1B
7s
SD, MagicBrush data
Subject-driven Image Generation
Textual Inversion (Gal et al., 2022)
1.1B
15m
Word embedding tuning Finetuning with preservation loss
DreamBooth (Ruiz al., 2023)
1.1B
10m
DreamBooth-Lowe (Hu al., 2021) SuTI (Chen al., 2023)
1.1B
8m
DreamBooth Low-Rank Adaptation
2.5B
30s
In-context + Apprenticeship learning
BLIP-Diffusion (Li et al., 2023a)
1.1B
8s
Pretrained encoder, Zero-shot
Subject-driven Image Editing
DreamFdit (Li et al., 2023b)
1.1B
8m
Dreambooth Region proposal
PhotoSwap (Gu et al., 2023) (Li et al., 2023a)
1.1B
7m
Dreambooth Cross- Attes-Atteniin Pretrained encoder, Zero-shot
1.1B
18s
Multi-concept Image Composition
CustomDiffusion (Kumari et al,, 2023)
1.IB
19m
Cross-attention updating
DreamBooth (Ruiz et al., 2023)
1.1B
11m
Finetuning with preservation loss
Textuallnversion (Gal et al., 2022)
1.1B
32m
Word embedding tuning
Control- guided Image Generation
ControlNet (Zhang Agrawala, 2023)
1.4B
8s
Zero convolution + Frozen model Multi-task pretraining. Zero-shot
UniControl (Qin et al., 2023)
1.4B
23s
Table 3: Overview of all the evaluated models and their parameter size and runtime. The models are listed in chronological order. The number of parameters and runtime for Dalle-3 and MidJoumnyy are unknown. MidJourney is not opensource and does not have a whitepaper in technical detail.
often misunderstood the prompt as "panda latte art". DALLE-3 and Midjourney understand most of the complex prompts than other models, and DALLE-3 is slightly better conflicting prompts.
Mask-guided Image Editing. We observed the outputs commonly contain obvious artifacts in the masked region boundaries for Stable Diffusion and GLIDE. While Blended Diffusion and Stable Diffusion XI. do not suffer from the same issue, they often produce unrecognizable outputs. Stable Diffusion XI. obtains the best results but the overall model performance is still far from satisfactory. Another common issue is that the filled regions can hardly harmonize with the background.
Text-guided Image Editing. One key requirement is to edit the image precisely and keep the back- ground untouched. This requirement is indeed challenging because the network has to understand of editing region from the semantic inputs. We discovered that Prompt-to-Prontt SDEdit, despite generating high-qu high-quality images, often result in completely different backgrounds. We also spotted that in many cases Tixt2Live will simply return the input as output, this phenomenon
Pix2PixZero. and
7

---

Published as a conference paper at ICLR 2024
Model
LPIPS +
CLIP t
SC
AQg
Overall
Fleisso
Kdo
Text-guided Image Generation
Dalle-3
N/A
0.2697
0.79++0.02
0.79- 1.
0.76++008
0.19
0.34
Midjourney
N/A
0.2839
0.67-00.06
0.92=00.66
0.73+0.07
0.34
0.31
DeepFloydIF
N/A
0.2814
0.65:10.02
0.62=0006
0.59:1002
0.32
0.51
Stable Diffusion XL
N/A
0.2886
0.62++0.03
0.64=00.05
0.59++0.03
0.37
0.61
Dalle-2
N/A
0.2712
0.58+00.4
0.62+0.066
0.54+0.04
0.27
0.40
OpenJourney
N/A
0.2814
0.53++0.02
0.59=00005
0.50+0.02
0.30
0.47
Stable Diffusion 2.1
N/A
0.2899
0.56:+0.02
0.53
0.50
0.38
0.50
Mask-guided Image Editing
-DXI.-Inainninng
0.15
0.2729
0.49++0.05
0.51=00.2
0.37++0.05
0.50
0.72
SD-Inpainting
0.21
0.2676
0.28:10.04
0.27
0.17::0.07
0.31
0.49
GLIDE
0.18
0.2578
0.20:10.05
0.48-00.06
0.16::0.05
0.33
0.56
BlendedDiffusion
0.33
0.2594
0.12:10.03
0.11=00.33
0.05:1:002
0.36
0.44
Text- Image Editing
MagicBrush
0.22
0.2675
0.51
0.65=00.66
0.47:10.02
0.44
InstructPix2Pix
0.32
0.2616
0.2911001
0.70=10.6
0.27:10.02
Prompt-to to-prompt
0.40
0.2674
0.17:00.5
0.55=0.09
0.15:10.06
CycleDiffusion
0.28
0,2692
0.17:10.03
0.56-0.11
0.14100.04
SDEdit
0.2872
0.04+10.33
0.56=0.12
0.04::0.03
Tex12Live
0.17
0.2628
0.02+10.01
0.82=0004
0.02+00.02
DiftEdit
0.22
0.2425
0.02++0.01
0.23=0004
0.01+:0.01
Pix2PixZero
0.60
0.2510
0.01++0.00
0.48+00.9
0.01++0.01
0.37
0.37
Subject-driven Image Generation
SuTI
0.77
0.2895
0.64
0.68=0.08
0.58++011
0.20
0.39
DreamBooth
0.77
0.2847
0.51++0.08
0.93=0.02
0.55+0.11
0.37
0.60
Diiffuiiiiuiin
0.77
0.2729
0.29++0.04
0.93+0004
0.35+0.06
0.22
0.39
Textuallnversion
0.81
0.2680
0.21
0.74=
0.21
0.35
0.52
DreamBooth-Lora
0.82
0.2988
0.07=00.1
0.82=0.07
0.09
0.29
0.37
Subject-driven Image Editing
PhotoSwap
0.34
0.2846
0.34++0.02
0.65
0.36+:0.02
0.35
0.46
DreamEdit
0.22
0.2855
0.31
0.61=10.03
0.32:+0.03
0.33
0.52
BLIP- Liff-Diffsioon
0.25
0.2901
0.09:10:33
0.70:10.02
0.09:10.03
0.41
0.47
Multi-concept Image Composition
CustomDiffusion
0.79
0.2929
0.26+10.01
0.86=
0.29:1001
0.73
0.88
DreamBooth
0.78
0.2993
0.11200.2
0.78=0002
0.13:0002
0.61
0.71
TextualInversion
0.80
0.2548
0.0410001
0.74=0005
0.0510001
0.62
0.77
Control-guided Image Generation
ControlNet
0.80
0.2555
0.42:10.05
0.19:10.04
0.23:10.04
0.37
0.57
UniControl
0.82
0,2604
0.38-10.07
0.20-1.06
0.2310007
0,36
0,58
Table 4: All the evaluated models from seven core tasks. Overall is the average of all VSC PQ. Fleisso and Kdg. denoting Fleiss' Kappa and Krippendorff's alpha for the overall average, respec- tively. We have more automated metric results in Appendix Table and correlations in Table 6.
also occasionally happened in other models. For paper claims, our evaluation ranking aligns with the findings from CycleDiffsuion, InstructPix2Pix. MagicBrush, Prompt-to-Promp and DiffEdit. We found our evaluation ranking does not align with Pix2PixZero, since their paper only tested on word-swapping examples, which not able to generalize to more complex edits.
Subject-driven Image Generation. Our evaluation results largely align with DreamBooth, BLIP- Diffusion, and SuTT findings. Specifically,, Textual inversion struggles maintain target subject fea- tures. DreamBooth can imitate subjects based images but occasionally resorts to copying learned images. DreamBooth- Lora struggles to generate desired subjects but can follow context prompts. BLIP-Diffusion can mimic target subject features but struggles with details. SuTI maintains high consistency with desired subjects and context, with tolerable artifacts some cases.
Multi-concept Image Composition. Our evaluations validate that CustomDiffusion is consistently better than the other two models. However, while learns the given multiple subjects' features
8

---

Published as a conference paper at ICLR 2024
better, it could fail to follow the prompts in many cases, especially actions and positional words. In contrast, DreamBooth learns the correct subjects some cases, and TextualInversion rarely learns the correct subjects. Interestingly, in some cases where DreamBooth does not learn the correct subjects, it could still follow the prompts correctly.
Subject-driven Image Editing. It is essential to modify the subject from the source the tar- get without causing excess changes to the background. Human evaluation is also conducted in DreamEdit for the comparison between Photoswap and DreamEdit, but our rankings differ due to varying evaluation criteria. PhotoSwap can adapt to the target subject from the source naturally in most cases but rarely preserves the background well. DreamEdit maintains the context most cases but sometimes leaves observable distortions at the edge of contextualization. BLIP-Diffusion fails the adaptation most of the time, compromising for a more realistic generation.
Control-guided Image Generation. Our evaluation shows there is no significant difference be- tween the two models in both automatic metries and human evaluation metrics. While UniControl also reported that there is no significant difference in automatic metrics, our human evaluation sults do not align. This can be due to the different evaluation standards and aspects. Nevertheless, it has come to our attention that neither of these models demonstrates high level of robustness. Scratches often appeared on the generated image.
5.2 ABLATION STUDY
Method of overall human score computation. We the overall score of the model as the ge- ometric mean of SC and PR score (i.e. O VSC x PQ). But we also explored the weighted x SC + x PQ, where both a and B are [0. 1]. We experimented and
sum setting O found that the weighted sum setting yields different ranking in the models. Take the text-guided image editing task as an example, as in Appendix Table 7, Text2Live outperforms CycleDiffusion in the weighted sum setting even though we found that CycleDiffusion performs better in the human examination. We investigated and found that majority of results in Text2Live simply return the input as output (in that case SC=0 and PQ=I). We tried adjusting the weightings but it still failed to reflect the actual performance of the model. Thus we decided to use the geometric mean setting to penalize the models.
Design choice of human evaluation metric range. When comes to human evaluation on a massive scale, it's essential create system that's easy understand and quick to use. In this investigation, we undertake an exploration into how different settings in this evaluation method can affect the results, showing in Appendix Table 8. Initially, our approach entailed the utilization of a range encompassing [0, 0.5, 1, 2] for both the Semantic Consistency (SC) score and the Percep- tion Quality (PQ) score, where means the image perfect. However, this configuration yielded suboptimal results in the Fleiss Kappa. Subsequently, an alternative configuration was employed, narrowing the range to [0, 1] for both the SC and PQ scores. This adjustment, while accommo- dating a binary classification, was observed to yield values that were overly polarized and extreme. To find the right balance between keeping values in a reasonable range and making sure the eval- uation method is reliable, resolved to define range of [0, 0.5, 1] while providing explicit and unambiguous guidelines.
CONCLUSION
In this paper, we propose ImagenHub as a continuous effort to unify all efforts conditional image generation into a library, easing access these models. We standardize the dataset and evaluation of these models to build our ImagenHub Leaderboard. We hope this leaderboard can provide a more reproducible and fair environment for researchers visualize progress in this field. A limitation of this work is the reliance on human raters, which is expensive and time-consuning. In the future, we plan to develop more generic automatic evaluation methods that approximate human ratings, helping people develop better models.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Our work aims to benefit the broader research community by providing standardized framework for evaluating conditional image generation models. For all the benchmarks, the ethical use of them. The datasets used our work are either publicly available or have been collected and curated exercising the utmost respect for privacy and consent. We will maintain leaderboard to track latest models and encourage open collaboration and discussion in the field. In human evaluation, we followed the minimum hourly wage of SI1. We also ensure that no personal information is collected and no offensive content is presented during human evaluations.
committed to
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European Conference on Computer Vision, pp. 707- 723, 2022.
Ali Borji. Pros and cons of gan evaluation measures: New developments, 2021.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wer Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. NeurIPS, 2023.
Guillaume Couairon, Jakob Verbeck, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2022.
deep floyd.ai. If by deepfloyd lab stabilityai, 2023. URL https:// com/ deep-floyd/Io floyd/IF.
Emily L Denton, Soumith Chintala, arthur szlam, and Rob Fergus. Deep genera- tive image models using laplacian pyramid of adversarial networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural In- formation Processing Systems, volume 28. Curran Associates, Inc., 2015. URL
https:
http::s//pxocceddnnng. htw.://eedddss..aiis.....eeeeeee cc http://pee__iileee//ee////0/ aal199995511bb554223333eeee32
pdf.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https:/ https///pppneeeeeeeeeoome net forummidumm PUIqjI4rzq7.
Joseph L Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and psychological measurement, 33(3):613- 619. 1973.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 01618, 2022.
Jing Gu, Yilin Wang. Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, et al. Photoswap: Personalized subject swapping in images. arXiv preprint 2305 18286, 2023.
10

---

Published as a conference paper at ICLR 2024
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference- frennce-freeeee evaluation metric for image captioning. In EMNLP, 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2021.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
Klaus Krippendorff. Computing krippendorff's alpha-reliability, 2011. URL https://api. semant iesaannacccolaar: wwt.o//////////oooooooooo pazg/CorrusID 59901023.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-concept
customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition, pp. 1931-1941, 2023.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to- text-to-image generation and editing, arXiv preprint arXiv:2305.14720, 2023a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In /CML, 2022.
Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing. arXiv preprint arXiv:2306. 1080066.2624 12624, 2023b.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
midjourney. Midjourney, 2023. URL https:/ /www mi
com hoon/.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 2022.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob
Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning, pp- 16784-16804, PMLR, 2022.
openai. Improving image generation with better captions, 2023. URL htthtt/s///dde openai...pp.......ooooia... com/ coompppper//ae dall-elllleeeeee.e.a. e-3. pdf.
openjourney.ai. Openjourney is an open source stable diffusion fine tuned model on midjourney images, 2023. URL https: http//hhggggaaaaeeeeeee/e wtt.eaacccccccoooeeeeeeeee
jow.eeerry./pjj............. co http///mmtttr////ceeoeoooe htrpg//ggaacccopoooppperrrrro//
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang. Yijun Li, Jingwan Lu, and Jun-Yas Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings, pp- 2023.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 2085-2094, October 2021.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Car- Niebles, Caiming Xiong, Silvio Savarese, al. Unicontrol: unified diffusion model for controllable visual generation in the wild. arXiv preprint 111. 2023.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/15, 1(2):3 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
runwayml. Stable diffusion inpainting, 2023. URL httph:ttp///c stable-dis di fusiion inpainting.
co
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 29, Cur- ran Associates, Inc., 2016. URL PRLLtttps//ppppceeiss....eerr / http:///0............222222222..
cttppppppccppppppeepeeeppeee files/
Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306. 00983, 2023.
stability.ai. Stable diffusion xl, 2023. URL
witeeedifffffiiooo ffusion.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-arr diffusion models. https:
//qithub. www.coo////a/a/ hww.o///gggggggaaa face///innaae///iaa htt.ggggggg.////.eeeeeeeeeeeee diffusers, 2022.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image genera- tive models. arXiv:2210. 11n.:2290..4896 14896 [cs], 2022. URL https: wwwpaaiivvv..o./.aa/.aaaa wtt.v/.rggabb////20.0..
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. Huggingface's transformers: Statc-of-the natural language processing. arXiv preprint arXiv:1910.03771,, 2019.
Chen Henry Wu and Fernando De la Torre. A latent space of stochastic diffusion models for zero- shot image editing and guidance. ICCV, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding. Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. NeurIPS dataset and benchmark track, 2023,
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
MORE METRICS RESULTS
Model
LPIPS L
DINO CLIP-11
DreamSim 1
FID 4
KID
CLIP
Text-guided Image Generation
Dalle-3
N/A
N/A
N/A
N/A
N/A
N/A
0.2697
MidJourney
N/A
N/A
N/A
N/A
N/A
N/A
0.2839
DeepFloyclIF
N/A
N/A
N/A
N/A
N/A
N/A
0.2814
Stable Diffusion XI.
N/A
N/A
N/A
N/A
N/A
N/A
0.2886
Dalle-2
N/A
N/A
N/A
N/A
N/A
N/A
0.2712
OpenJourney
N/A
N/A
N/A
N/A
N/A
N/A
0.2814
Stable Diffusion
N/A
N/A
N/A
N/A
N/A
N/A
0.2899
Mask guided Image Editing
SDXL- Inpaannng
0.15
0.90
0.94
0.15
59.04
-0.0033
0.2729
SD-Inpainting
0.21
0.85
0.91
0.20
86.23
-0.0008
0.2676
GLIDD
0.18
0.91
0.94
0.14
61.71
-0.0028
0.2578
BlendedDifffssoon
0.33
0.80
0.87
0.26
102.73
-0.0012
0.2594
Text-guided Image Editing
MagicBrush
0.22
0.88
0.92
0.19
77.81
-0.0027
0.2675
InstructPix2Pix
0.32
0.77
(1.66
0.30
100.30
-0.0014
0.2616
Prompt-to- ceappptto-proooe to-prompt
0.40
0.68
0.81
0.36
137.68
0.0003
0.2674
CycleDiflusion
0.28
0.79
0.88
0.27
108.29
-0.0008
0.2692
SDEdit
0.61
0.57
0.76
0,50
156.12
0.0025
0.2872
Text2Live
0.17
0.87
0.92
0.18
68.33
-0.0028
0.2628
DiffEdit
0.22
0.78
0.85
0.31
121.79
0.0069
0,2425
Pix2PixZero
0.60
0.53
0.76
0.52
161.64
0.0072
0.2510
Subject-driven Image Generation
SuTI
0.77
0.69
0.81
0.32
164.97
0.0111
0.2895
DreamBooth
0.77
0.61
0.80
0.40
173.71
0.0057
0.2847
BLIP-Diffusion
0.77
0.63
0.82
0.36
166.53
0.0102
0.2729
Textuallnversion
0.81
0.37
0.65
0.61
229.18
0.0122
0.2680
DreamBooth-
0.82
0.36
0.70
0.61
234.95
0.0125
0.2988
Subject-driven Image Editing
PhotoSwap
0.34
0.65
0.83
0.37
130.81
0.0009
0.2846
DreamEdit
0.22
0.74
0.87
0.28
118.95
0.0011
0.2855
BLIP- FLIf-DDifussoo
0.25
0.80
0.89
0.23
102.17
0.0007
0.2901
Multi-concepl Image Compusition
CustomDiffusion
0.79
0.50
0.70
0.55
207 /43
0.0341
0.2929
DreamBooth
0.78
0.39
0.64
0,66
257.60
0.0533
0,2993
TextualInversion
0.80
0.44
0.66
0,65
226.25
0.0491
0.2548
Control guiieed Image Generation
ControlNet
0.80
0.43
0.70
0.56
219.29
0.0330
0.2555
UniControl 0.82
0.42
0.68
0.58
222.57
0.0192
0.2604
Table 5: All automated metrics for the evaluated models based on our benchmark dataset.
13

---

Published as a conference paper at ICLR 2024
PQT
SC
corr(LPIPS)
com((DIN)
con(CLIP-I)
con(DreamSim)
corr(CLIP)
Text-guided Image Generation
N/A
N/A
N/A
N/A
-0,0819
Mask-guided Image Editing
-0.0502
-0.0421
+0,0308
-0.0294
-0.0257
Text-guided Image Editing
0.0439
0.0449
0.0432
0.0437
Subject-driven Image Generation
0.0905
0.1737
0.1204
0.0943
-0,0117
Subject-driven Image Editing
0.2417
0.1690
0.3770
0.3846
-0.0443
Multi-conccept Image Composition
-0.1018
-0.1657
-0.0965
0.0140
-0.0107
Control-guided Image Generation
0.1864
0.1971
-0,0985
-0.0045
-0,2056
Table 6: Metrics correlation. we inverted the signs for metric that is the lower the better. The computable metrics all hold different assumptions which only contribute part of the aspects in human evaluation. This makes low correlation with human evaluations.
O VSC x PQ
= 0.5SC
+ 0.5PQ
O 0.7SC +
0.3PQ
Setting
Osum
OAva
O5a
OAwg
O5um
OAva
MagicBrush
83.51
0.47
103.75
0.58
98.85
0.55
CycleDiffusion
24.89
0.14
65.25
0.36
51.28
0.29
DiffEdit
1.71
0.01
22.08
0.12
14.38
0.08
Text2Live
4.08
0.02
75.25
0.42
46.75
0.26
Table 7: Comparison overall human score computation setting.
categories [0, 0.5,
categories [0, 0.5, 1,
categories [0, 1]
Setting
O Fleisso
Kdo
Oaeg
Fleisso
Kdo
Onw
Fleisso
Kdo
BLIP-Diffussee
0.13
0.41
0.47
0.09
0.29
0.50
0.05
0.40
0.40
DreamEdit
0.33
0.52
0.32
0.26
0.55
0.21
0.38
0.38
PhotoSwap
0.62
0.35
0.46
0.36
0.25
0.50
0.29
0.37
0.37
Table 8: Ablation study to understand the impact of the granularity of SC and PQ.
14

---

serious some Published as a conference paper at ICLR 2024
A.2 HUMAN EVALUATION GUIDELINE
To standardize the conduction of a rigorous human evaluation, we stipulate the criteria for each measurement follows:
Semantic Consistency (SC), score in range [0, 0.5, 1]. measures the level that the generated image is coherent in terms of the condition provided (i.e. Prompts, Subject Token, etc.).
Perceptual Quality (PQ), score in range [O, 0.5, 1]. It measures the level at which the generated
image is visually convincing and gives off natural sense.
Meaning of Semantic Consistency (SC) score:
Image not following one more of the conditions at all (e.g. not following the prompt at all, different background editing task, wrong subject in subject-driven task, etc.)
SC=0
SC=0.5 all the conditions are partly following the requirements.
SC=1 The rater agrees that the overall idea is correct.
Meaning of Perceptual Quality (PQ) score:
PQ=0: The rater spotted obvious distortion or artifacts at first glance and those distorts make the objects unrecognizable.
PQ=0.5: The rater found out the image gives off an unnatural sense. Or the rater spotted spotted some minor artifacts and the objects are still recognizable.
PQ=1: The rater agrees that the resulting image looks genuine.
Raters have strictly adhere to Table when rating.
Condition
Condition (if applicable)
Condition (if applicable)
SC rating
no following at all
Any
Any Any
Any
no following at all Any
0
Any
following all following some or most part following some most part following some part following most part
0
following some part following some most part following some part or more following most part
following some or most part following some part following some or most part following most part
0.5
0.5
0.5
Objects in image Unrecognizable
Artifacts
Unusual sense
PQ rating
Any
Recognizable
Any
0.5
Recognizable
Any
some
0.5
Recognizable
none
little None
Table 9: Rating guide checklist table.
Artifacts and Unusual sense, respectively, are:
Distortion, watermark, scratches, blurred faces, unusual body parts, subjects not harmonized
wrong sense of distance (subject too big too small compared to others), wrong shadow, wrong lighting, etc.
15

---

Published as a conference paper at ICLR 2024
A.3 RATING EXAMPLES
There are some examples when evaluating:
Texee Generation.
outputA
outputB
outputC
"prompt": "One cat and two dogs sitting on the grass.", "category": "Counting".
Dep. Deeaing
D THOOI
ITAAY
DEEP LEARNING
DEEP
LAARING DEEFF
"romptt": "A sign that says 'Deep Learning' :
"category": 'Text"
Figure 5: Rating examples on Text-to-Image Generation task.
OutputA1: [1, 0.5]. SC=1: Prompt perfectly align. PR=0.5: Minor distortion on the cat's facial features.
10.5, 5, 0.5]. SC=0.5: 3 dogs appeared instead of cat and dogs. PR=0.5: Minor distortion was found on the animal's face and the watermark.
OutputB
OutputCi: [1, 0.5]. SC=1: The prompt match perfectly with the image. PR=0.5: Minor
distortion the dog's facial features.
OutputAg: [0.5, 1]. SC=0.5: A sign appeared, but failed spell the word. PR=1: The image look generally real but with some lighting issues.
OutputB2: [0.5, 5, 0. 0]5]. SC=0.5: A sign appeared, but failed spell the word. PR=0.5: The background looks so unnatural.
OutputCa: [0.5, 0]. SC=0.5: A sign appeared, but failed to spell the word. PR=0: Heavy distortion on both text and strong artifacts the background.
16

---

Published as a conference paper at ICLR 2024
Mask-guided Image Editing.
input
mask
outputA
outputB
"source glwbal_cappiie "A small black dog playing with a frisbee.", "instruction": "turn the frisbee into soccer ball", globall caption": : "A small black dog playing with soccer ball."
"source glwballlcoppeeeeee "A Zebra standing in between group of large rocks", "instruction": "Make the zebra regular horse.", : 'A horse standing in between group of large rocks"
Figure 6: Rating examples Mask-guided Image Editing task.
OutputA,: [1, 0. 5]. SC=1: Clearly generate the soccer. PR=0.5: The edit region does not blend well with the context.
OutputB;: [1, 1]. SC=1: Successfully add soccer. PR=1: The soccer is naturally blended
with the context.
OutputAg: [o, 0]. SC=0: Generated content can not be regarded horse. PR=0: The middle left part is not natural.
OutputB2: [0.5, 0]. SC=0.5: The object is horse-like but not good. PR=0: The whole image not natural.
17

---

Published as a conference paper at ICLR 2024
Text-guided Image Editing.
input
outputA
outputB
"source Sooourc_gllobl... caption": "A bull on a farm walking around pen.", "instruction": "Have the cow a hat.", ", "target- caption": : "A stylish cow wearing a walks around pen on a farm.'
"source_ Brource_globbl.... "A skateboarder is doing trick on hand rail." " "instruction": "What the man hat?", "target glliillooooaooeeeeo caption": "A skateboarder with a hat is doing a trick on a hand rail."
Figure 7: Rating examples Text-guided Image Editing task.
OutputAj: [0, 0]. SC=0: The image does not follow the instruction at all. PR=0: Heavy
distortion the cow's facial features.
OutputB;: [1, 1]. SC=1: The hat gives a nice specular reflection. PR=1: It looks real.
OutputAg: 1, 0.5]. SC=0.5: The hat exists but does not suit well. PR=0.5: The impor- tant object look distorted.
OutputB2: [0, 0]. SC=0: The background completely changed. PR=0: The whole image look distorted.
18

---

Published as a conference paper at ICLR 2024
Subject-driven Image Generation.
input
outputA
outputB
"subject_id" 16, "subject": "dog8",
"prompt": "A <token> dog swimming a river."
"subject id": 1,
"subject': "backpack_dee dog',
prommpt": 'a man carrying <token> backpack dog."
Figure Rating examples Subject-driven Image Generation task.
OutputA,: [1, 1]. SC=1: The output does match the subject and prompt. PR=1: looks real.
OutputBi: [0, 0]. SC=0: The subject dog missing. PR=0: Serious unusual body.
OutputAg: [0, 0.5]. SC=0: The output does not match the subject. PR=0.5: Some unusual
sense on the backpack.
OutputB,: [O, 0. 0.5]. SC=0: The output does not match the prompt. PR=0.5: Some unusual
sense the desk with the grass.
19

---

Published as a conference paper at ICLR 2024
Multi-concept Image Composition.
input
outputA
outputB
'flower in the wooden pot on a table."
"target_caption": 'chair with a screaming cat sitting on it."
Figure Rating examples on Multi-concept Image Composition task.
OutputAi: [1, 1]. SC=1: Both subjects accurately represent the intended subjects and the prompt actions match. PR=1: In general the image natural and as real.
OutputB;: [0, 1]. SC=0: Subjects
correct even though prompt action matches. PR=1:
The image looks real.
OutputAg: [0.5, D.5]. SC=0.5: Both subjects accurately represent the intended subjects and but the prompt action doesn't match. PR=0.5: There is human face the chair so the important subject chair looks unrealistic, but do not strongly detract from the image's overall appearance.
OutputBy: [0, 0.5]. SC=0: Subject chair missing. PR=0.5: Minor distortion the cat body and background.
20

---

Published as a conference paper at ICLR 2024
Subject-driven Image Editing.
input
token
outputA
outputB
"subject": "colorful_se '
9
"subject": "clock"
Figure 10: Rating examples on Subject-driven Image Editing task.
OutputA1: [0.5, 0.5]. SC=0.5: Some details of the subject do not match the token. PR=0.5: Some unnatural sense on the hand features.
OutputB;: [0.5, 5, 11. SC=0.5: The subject does match the token, but removing the hand is unnecessary edit. PR=1: It looks real.
OutputAg: [0, 0. 5J. SC=0: Subject clock does not match. PR=0.5: Some distortion on the numbers of the clock.
OutputBy: [0, 0. 0] SC=0: Subject clock does not match. PR=0.5: Some distortion on the numbers of the clock and the edit region does not blend well with the context.
21

---

Published as a conference paper at ICLR 2024
Control-guided Image Generation.
input
outputA
outputB
"prompt": "golden gate bridge sunset, Golden Gate Bridge San Francisco, USA "control_type" 'hed".
"text": "a man riding skateboard the side of a ramp", ccontrol_type". "depth".
Figure 11: Rating examples on Control-guided Image Generation task.
The generated image perfectly described all the required at- tributes of the user prompt, and even work. PR=0.5: There are some missing details in the background. The details in the far-side mostly blurred unnaturally.
OutputAj: [1, 0.5]. SC=1:
OutputB1: [1, 0.5]. SC=1: The generated image perfectly described all the required at- tributes of the user prompt. PR=0.5: The details of golden gate bridge are mostly blurred unnat- urally.
OutputA2: [0, 0]. SC=0: The output does not correspond to the controlled depth image. PR=0: Heavy distortion on the background and the man.
OutputBy: [0, 0]. SC=0: The meaning of the cannot be obtained from the output. PR=0: Heavy distortion.
22

---

Published as a conference paper at ICLR 2024
A.4 DATASET INFORMATION
Dataset Details.
Dataset Distribution.
28
Arilaca
Aniaass
Adt
People
People
Artimals
Outcoo Stenes
Scones
Word Krowlegge Food
Plants
#llsttratoon
Plarte
Word Krowledge
(a) Text-to-Image Image Generation dataset
(b) souided/Text-guideee Image Editing dataset
21
10
Animal
iBustradors
Attloss
Artlecta
Artimate
Food
(c) Subject-driven driven Image Generation dataset
(d) Multi-concept Image Composition dataset
119
People
Indoor
Antmats
Krowledge Food
Venicies
*Attiectt Illaadssions Ariimade Food
Arts
Tlustration
Plants
Subject-driven Image Editing dataset
(f) Control-guided Image Generation dataset
Figure 12: Objects presented in each task's 's prompt text.
23

---

Published as a conference paper at ICLR 2024
A.5 VISUALIZATION
DALLE
DeepFloydIF
OpenJourney
SD
SDXL
A panda making latte art.
dogs the
giraffe underneath
SAREPRAM
ceniiiitttn smafml vessef epropoeilled ors, sauls, engle.
A beautiful photorealistic anime illustration of urbex industrial architecture architecture unfinished building abandoned post office renzo piano, laser extraterrestial sunset lake vaporwave elysian at night reclaimed by nature magic realism myst wilderness, archdaily, wallpaper, highly detailed, trending artstation.
Figure 13: More samples from our Text-guided Image Generation dataset and ImagenHub outputs.
24

---

Published as a conference paper at ICLR 2024
Input
mask
BlendedDiffusion
Glide
SDinpaint
SDXLInpaint
eating park." Taaget_global__oe caaballcaption": wearing eating food
caption":
"instruction: "Heve the man be weering kilt"
"source caapiinn "Three zebras standing middle plain with majestic mountains background. "inttruttin"" "put zebras next to river"
pargot_gtobal caption": "Three zebras standing next rivor with majestic mountains in the background."
Saarce_plobel_eeee Leaption": : "A white plate topped with sandwich and rice." "instruction: pizza sandwich." "target_ global_ caption": white plate topped with pizza and rice."
Soarce_llobll__c Leaption:: !: "A kitchen filled with waoden floors and a stove top oven." "instruction': "get rid of table" "target global caption": kitchen filled with woaden floors and stove top oven, minus the vase on top of the table."
"source plobal. caption': : "Car parked parking lot front of builcing."
"instruction: "edit the background by removing the and placing castle" castle parking
Langet_global_ceee jubballcaption": "Car parked
Figure 14: More samples from our Mask-guided Image Editing dataset and ImagenHub outputs.
25

---

Published as a conference paper at ICLR 2024
Oyeleiiflusion
MagicBrush
Ps2PsZZero
Promp12rrmpe
SDEdit
"ourre giibal exprion" Mamn making pacly parxe "mttrutinn" "joce man glacue" "trget gobal.oooooooo
making goofy with
yourne plobol caption" grown baying and steppng top of
Tratrctionn "Chngg cal
togetaabbd coption:: grown laying down slceping tn
"sourte plbbi copplon" "A plece of ple has tona ros and whipsed cream sunounding it on white plate
Tretruction "pur strabbrrr on sime" targel_ghhal gitbal cxption" piace of with barurtan, whipped cream strowbertte surmunddng on white plate.
Youret sourt_dlbo plobo coppbo" "A wrize wall mounted ove sitiinggg non to stove"
Tratnuctoon "Lt water un thom fucee"
torget_gibil_ coploo" whit vall mounted meen diting next stowe with water running from the fuwen!
Touuno, a roall exption:: 'A men riding a bibe on beach neat to the nowes.
"ntraction:: "Add crune dhp to the ozeen." "trget_gion globol caption". man rddng bike ona beach noxt to the ocean with crube ship the tockground""
Figure 15: More samples from our Text Text-guided Image Editing dataset and ImagenHub outputs.
26

---

Published as a conference paper at ICLR 2024
input
BLIPDiffusion_ Gen
DreamBooth
DreamBoothLora
SuTI
Textuall nverssio
"suaject"; "dog?",
"prompt: 'a staken- dog reading book with pink glasses on"
GRIAN
"subject"; "re_car",
"rooptt: la stolenz- re car driven by the super mario"
Bear Errginl
"subject":
"berry_ bowl",
"prompt": berry bowl floating on
"subject":
"poop, eejji".
"prorpt":: <tokent poop emoji wearing sunglasses*
C
"sujject":
"teapot".
"prompt": day stoken> teapot sitting glass table"
Figure 16: More samples from our Subject-Driven Image Generation dataset and ImagenHub out- puts.
27

---

Published as a conference paper at ICLR 2024
input
token
BLIPDiffusio Edit
DreamEdit
PhotoSwap
"subject": "pink_ sunglasses"
"subject": sloth "oothhelushie"
"subject": "duck_toy"
"subject": "fancy_boot"
"subject": "dog"
Figure 17: More samples from our Subject-Driven Image Editing dataset and ImagenHub outputs.
28

---

Published as a conference paper at ICLR 2024
input
CustomDiffusinn
DreamBooth
Textuallnversion
"prompt": "the cat playing with wooden pot in garden", "concept1"; "wooden pot",
"concept2"; "cat"
"prompt": "Oil painting of a teddybear in front of the barn", "concept1": "teddybear",
"concept2": "barn"
"prompt": "a digital illustration of flower in wooden pot", "concept1": "wooden pot",
"concept2": "flower"
"prompt": "Watercolor painting of tortoise plushy next to cat", "concepti": "tortoise plushy",
"concept2": "cat"
10
DI serteeatt conccaa www.eaeeeeeeee..o
9O19 60GOO Be
"prompt": "Vintage poster of car with dog in the backseat", "concept1": "dog",
"concept2": "car"
Figure 18: More samples from our Multi-Concept Image Composition dataset and ImagenHub out- puts.
29

---

Published as a conference paper at ICLR 2024
input
ControlNet
UniControl
Golden gate bridge san francisco, Golden Gate Bridge San Francisco, USA
Closeup a brown bear sitting in grassy
"New Year's Eve: Dubai Will Attempt ""Largest Fireworks Display" World Record"
A yellow fire hydrant sitting in the grass
man standing on a mat with pair dumbbells, deep fake, youtube video screenshot, off the shoulder shirt, underbrush wash, two legs two arms one head, neck shackle, grainy footage, surfaces blemishes, artem, uses c4, buttshape, sconces, threes, head to waist
Figure 19: More samples from Control-guides Image Generation dataset and ImagenHub out- puts.
30

---

fully under- Published as a conference paper at ICLR 2024
1.0
Mean Scores
0.9
Krippendorff's Alpha
Values
0.8
0.7
0.70
0.71
0.70
0.70
0.64
0.6
0.5
3
4
7
9
10
Number of Raters
Figure 20: Mean scores of overall human rating and Krippendorff's Alpha for each number of raters model.
A.6 HUMAN EVALUATION PROCEDURE
Prolific. In this work, we used the Prolific platform receive human feedback on rating AI- generated images. We applied the following filter for worker requirements when creating the project: 1) The Worker's primary speaking language has to be English. 2) Worker must complete rating one set of images (around 150-200 images) in go. We do that ensure workers stand the rating instructions, and each set of image rated by the same person so we can perform Inter-rater Reliability Analysis across the whole of the benchmark. Workers will rate each image in [SC, PQ). while SC in [0, 0.5, I] and PQ [0, 0.5, I]. Instructions and images are hosted on website. Workers are provided with detailed instructions and rating examples to assist their rating process. Workers will submit the responses in a tsv file. To validate the annotation quality, we se- lected few (around 10-20) significant samples with obvious ratings for each task. We rated them by ourselves and then used them as reference to determine the performance of annotators. Poor performance will be rejected.
Sample Size Consideration of Human Raters.
We studied the effect on the number of human raters. recruited 10 human raters on Prolific to rate 15 samples generated using the Midjourney model. In Figure 20, the mean scores seem to remain relatively constant across the number of raters, with a slight increase 4 raters and the highest mean score at 5 raters. The values for the mean scores are hovering around 0.70, as the error bar shows the standard deviation increases. the number of raters increases. It starts at around 0.70 with raters and has noticeable drop at 10 raters to 0.64. We can observe that the mean score will be always consistent, while the reliability of that score measured by Krippendorff's Alpha decreases slightly as raters involved, as more variance is introduced. To make our human evaluation protocol efficient, we picked 3 as the number human raters.
the other hand, Krippendorff's Alpha generally decreases as
31

---

Published as a conference paper at ICLR 2024
A.7 VISUALIZATION TOOLS (IMAGEN MUSEUM)
Experiment Transparency. We are hosting the benchmarking results publicly. Thus showing the true performance of each model. Viewers gain their insights from the result by looking the actual results. Our framework encourages future researchers to release the human evaluation generation results publicly. By standardizing the human evaluation protocol, human evaluation results would become far more convincing with the experiment transparency.
Vinaaiza Expertuaan:: Inaggattaa Text Galded
Figure 21: The webpage design of Imagen Museum (Text-Guided Image Generation page).
Viaualie Experimont: InsggelHok. Teai-Galeri_ Geserated on ENTIEEES 80:46:50
Figure 22: The webpage design of Imagen Museum (Text-Guided Image Editing page).
Please refer http://sghhhlloooooCcoooaaaaCCee for the visualization results.
32

---

Published as a conference paper at ICLR 2024
A.8 ADDING NEW MODELS OUR FRAMEWORK.
Third parties are welcome to use our provided tools and frameworks for their work. Here is quick walkthrough of how third-party member extend the framework support their model's inference. We use the Text-Guided Image Generation (Text-To-lmage) task as an example for the sake of simplicity.
The infermodel class is designed to have the following two methods:
-init_(args) for class initialization.
infer_one_image (args) to produce image output. Please try to set the seed as 42.
In that case, you will add new file in Shageen.hub/inferrooeell imagen.hub/inee www.uu/iiffromeeii.. mow.i//ese/eessmmmmmmee.......
folder.
Then line can be added in imagen_uub/
And modify the template config.ym yml file and add your own model.
Finally run
Please refer http://wwweeeeecccccccccoooeeiii html for details.
33

---

Published as a conference paper at ICLR 2024
A.9 IMAGENHUB AS AN INFERENCE LIBRARY.
Custom inference of multiple models.
Figure 23: Image generated in this custom inference example.
Evaluate images with autometrics.
Figure 24: Image evaluated in this autometrics example.
Please refer http://nweaaalllaallcccccccceoce for details.
34


Reviews:
[{"review_id": "GT5pRnRJwd", "reviewer": "Reviewer_nZYf", "summary": "The paper introduces ImagenHub, a standardized framework for evaluating conditional image generation and editing models, addressing inconsistencies in experimental conditions. It defines key tasks, creates evaluation datasets, establishes a unified inference pipeline, and introduces human evaluation scores. Results indicate that existing models generally perform poorly, except for Text-guided and Subject-driven Image Generation, and validate most claims from published papers while highlighting the inadequacy of existing automatic metrics, with plans to continue evaluating new models and tracking progress in the field.", "questions": "Q: Fig 2 -> what does y axis show? No label\n\nQ: Regarding the ImagenHub dataset: it seems like you are using data that is already been used by others. What is some researchers have already used this data to tune their models? Couldn’t you collect an independent new test set?\n\n\nQ: ImagenHub Inference Library is a great job. How you ensured that the best parameter setting is chosen to generate best results for each model?\n\n\nQ: What is the last row of Fig 3 showing?!\n\n\nQ: In Eq. 1, why min is used? Isn’t min too stringent here? Why not mean instead!?\n\nQ: why are there multiple errors bars for each condition? Not clear\n\nQ:  “We assigned 3 raters for each model and computed the SC score, PQ score, and Overall human score”. 3 subjects is really not that many here and this makes the results less reliable.\n\nQ: what is the keyword column in Table 3?\n\nQ: How is the overall column is computed in table 4?\n\nQ: No information about the subjects and biases etc are given.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This is an extensive endeavor! Comparison of several models plus human evaluation is presented. It is an important problem and the this is a timely study. In general, I am leaning towards accepting the paper but there are several issues and questions that need to be addressed. I would like to see the authors responses first.", "weaknesses": "A major contribution is human judgment which has some issues. First, the number of subjects is small, Second, details of how experiments are conducted and information about them is missing. \n\n\nWriting can be improved.\nTypos here and there:\nOne of the most popular task —> tasks  [page 1]\nWe found that evaluation results from the published papers from are generally [page 3]\nThese methods rely on the statistics on an InceptionNet pre-trained on the ImageNet dataset. [page 4]\nA limitation in this work is the reliance on human raters, which is not only expensive and time-consuming. [page 9]\n\nPage 3 “The goal of conditional image generation is to predict an RGB image” —> I think predict is not the right word here", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "02o2lo7ibz", "reviewer": "Reviewer_gWLk", "summary": "This paper presents ImagenHub, a dataset and library for standardized evaluation of conditional image generation models. A large amount of models is evaluated using a unified evaluatoin protocol of human raters. Two metrics are proposed to judge semantic consistency and perceptual quality, and the evaluation protocol is adjusted for high inter-worker agreement.", "questions": "- Is it possible to analyze drift of user ratings over time? In other words, how much is the rating influenced by the experience/exposure of a rater to the evaluation platform?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper tackles an important problem, namely inconsistent evaluation protocols of the large amount of recent image generation methods.\n- The paper presents a sound approach for fair comparison using human raters.\n- The paper contributes a library to standardize and ease the evaluation of future generative models.", "weaknesses": "- The paper states that 83% of the published results are consistent with the ranking, and the presented evaluation results often validate the results of published works.\n  - a) Where does this 83% come from?\n  - b) What about the other 17%? What kind of results from published work is not consistent with the presented work? Is it due to limitations of the presented paper or wrong claims by published work?\n\nTwo minor points:\n- It would be beneficial to incorporate more automatic measure such as the commonly used FID or detection based scores to evaluate spatial fidelity, object recognizability as well as counts of objects.\n- It could be interesting to see an analysis on the costs and time needed of such a unified evaluation protocol given that the method relies on human raters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VZwdro7ogK", "reviewer": "Reviewer_JKzK", "summary": "This paper proposes ImagenHub, a new benchmark for conditional image generation based on human evaluators.\nThe benchmark evaluates seven conditional image generation tasks. \nDuring evaluation, human evaluators will follow two major metrics, namely semantic consistency and perceptive quality. \nThe formal metric (SC) ensures that the generated image is aligned with the given condition, while the second metric (PQ) ensures that the generated image is of good visual quality.\nThe authors evaluate major opensource image generation approaches based on the two metrics and report the results.", "questions": "1. How do the authors validate the annotation quality, and the measure the performance of annotators?\n2. How much human labor is required to build this benchmark (in annotator x hour)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper first proposes a comprehensive benchmark to evaluate different conditional image generation tasks. It evaluates a bunch of image generation models with human evaluators, and provides comprehensive evaluation results to the community.", "weaknesses": "1. It would be better if the authors can involve more top-performing image generation approaches in this comparisons (though some of them may not be opensource), like MidJourney or DALLE-3 (in a later revision of the paper). \n2. Some previous works (like T2I CompBench) have also proposed some evaluation metrics for benchmarking conditional image generation models (related to the Semantic Consistency part in this paper). It would be better to discuss and see if such metrics result in similar trends compared to the human-based evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0U76CgQ4gL", "reviewer": "Reviewer_teCT", "summary": "Paper proposes a library for the evaluation of coditional image generation models. They consider 7 tasks (text-guided, subject-driven, control-gruided etc). They define two human evaluation scores (semantic consistency and perceptual quality), train the raters, and evaluate around 30 models for the various tasks.", "questions": "I think the study is of interest for many people. However, I found the technical contribution still a bit shallow and the possible usage for future model evaluation unclear. If some results on weakness point 1 could be added to the paper, I would probably be willing to raise my score.\n\n- please address the mentioned weaknesses. \n\nminor remarks:\n- ref to Table 10 in section 4.2 is wrong (should be table 2)\n- spellcheck 'Inference: ' text on page 2.\n\n-------------------------------\nPOST REBUTTAL:\nI thank the reviewers for their feedback. I apologize for mising Table 8 in Appendix. And I appreciate the effort done to ensure ongoing usefulness of the proposed benchmark for future users (A7-A9). I have raised my score.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. paper is well written and presentation is good. \n2. a fair comparison based on human raters is interesting for many users and scientists in this dense research field. \n3. evaluation setup and comparisons are well-designed and seem fair (many methods use their own curated datasets and are here compared on the same data).", "weaknesses": "1. I think the paper would have been stronger if the authors would have directly compared the human evaluations with existing automatic evaluation metrics. It would be very interesting to know the correlations. \n\n2. Ideally we would have computable metrics which correlate high with human evaluations. The paper does not explain very well what the main problems are of existing metrics. It would also be interesting to see what parts of human evaluation are missed by currently used metrics.\n\n3. It remains unclear how a third party with a new method could make use of this benchmark since it is based on human raters. It would be nice if there is some rater-training guide which would allow other researchers to also evaluate their method on the proposed benchmark. There are no safeguards for the maintenance of the benchmark by the authors.\n\n4. I found the discoveries and insights not especially surprising. They were often more based on looking at results than referring to the human evaluation rates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)