DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation

## Abstract

Abstract Personalized image generation holds great promise in assisting humans in everyday work and life due to its impressive function in creatively generating personalized content.
However, current evaluations either are automated but misalign with humans or require human evaluations that are time-consuming and expensive.
In this work, we present DreamBench++ , a human-aligned benchmark automated by advanced multimodal GPT models.
Specifically, we systematically design the prompts to let GPT be both human-aligned and self-aligned, empowered with task reinforcement.
Further, we construct a comprehensive dataset comprising diverse images and prompts.
By benchmarking 7 modern generative models, we demonstrate that DreamBench++ results in significantly more human-aligned evaluation, helping boost the community with innovative findings.

## 1 Introduction

Driven by the significant advances in large-scale text-to-image (T2I) generative models [, , , , , , , , , , , , , ], it is now possible to generate images conditioned on not only arbitrary text prompts but also by given reference images—personalized image generation [, , , , , , , , , , , , , , , , , , , , ].
In general, to be useful as an artistic creation tool for inspiration or products [], the following two basic criteria must be fulfilled:
i) Prompt following (image & prompt consistency).
Generated images must follow the prompt description, which is a requirement shared with vanilla T2I generation [, ].
ii) Concept preservation (image & image consistency).
For personalized image generation, the concept of the reference image, i.e., the main subject’s semantic details (e.g., facial characters) or high-level abstractions (e.g., overall style), must be preserved in the generated image.
For example, a user may want to “imagine his own dog traveling around the world” [], and the generated dog must be the same as his but traveling.

Figure: Figure 2: Qualitative comparison of concept preservation evaluation between DreamBench++ and traditional DINO []. DINO often fails to yield human-aligned evaluation while our DreamBench++ succeeds using multimodal GPT models as the evaluator.
Refer to caption: x2.png

To meet the aforementioned requirements, numerous efforts have been devoted.
One line of fine-tuning-based works focuses on fine-tuning general T2I models to specialist personalization models by reproducing specific concepts present in training sets [, , , ].
Meanwhile, another line of encoder-based works, instead, achieves concept-preservation by training features adaptation to inject reference image features into a general T2I model [, , , , , , ].
Although remarkable results have been achieved, one question arises: can we comprehensively evaluate these models to figure out which technical route is superior and where to head?

In this work, we aim to answer this question by developing a new benchmark that properly evaluates personalized T2I models driven by the above two requirements.
We present DreamBench++, a comprehensive benchmark designed based on the following de-facto principled advantages:

- 1.
Human-Aligned
As shown in Fig. 2, traditional metrics like DINO [] and CLIP [] often result in significant discrepancies from humans.
This is caused by the image similarity measurement nature of DINO and CLIP models, and thus crowd-sourced human evaluation is typically necessary for obtaining a correct quantitative understanding of generated images [, , ].
Therefore, different from existing works that utilize CLIP and DINO as metrics that may be humanly misaligned, our DreamBench++ demonstrates surprisingly consistent evaluation results aligned with humans.
For instance, by evaluating 7 modern models, DreamBench++ achieves 79.64% and 93.18% agreement with human’s evaluation in concept preservation and prompt following capabilities, respectively.
Notably, it is +54.1% and +50.7% higher than traditional DINO and CLIP metrics.
- 2.
Automated
However, it is non-standardized and expensive to perform high-quality human evaluations, making human evaluations non-trivial to use.
To address this challenge, DreamBench++ achieves automated but human-aligned evaluation by using advanced multimodal GPT models as metrics such as GPT-4o [].
The challenges lie in two aspects: i) prompt design and ii) reasoning procedure for scoring.
We systematically standardize the automated GPT evaluation by first designing the evaluation instructions that provide overall task requirements, where language is a general interface for instructing human preference.
Inspired by Self-Align [], we instruct the GPT model to conduct internal thinking that aligns itself for better task and preference understanding.
Then, the GPT model provides the summary & planning for the task and scoring criteria, and the final scores are provided with optional
chain-of-thought (CoT) [, ].
- 3.
Diverse
To avoid biased understanding due to the low diversity of evaluation data,
DreamBench++ collects a large number of diverse images covering various levels of difficulty categories, from animals and styles that are relatively simple to more challenging human subjects, objects, and non-natural image styles (See Fig. 1).
Compared to commonly used DreamBench [] that only contains 30 subjects and 25 prompts, DreamBench++ scales up the benchmarking data to 150 images and 1,350 prompts, i.e., 5$\times$ and 54$\times$ more than DreamBench.
As a result, when evaluating models on this larger dataset, different but more robust conclusions have been obtained that demonstrate an unbiased and more comprehensive evaluation of DreamBench++.

Takeaways  Here, we offer some insightful findings that we obtain
by evaluating 7 modern personalized T2I models:
i) DINO-based rating puts tremendous weight on resemblance in overall shape or color and neglects detailed visual features, which makes it a sub-optimal metric for personalized image generation evaluation;
ii) The main goal of personalized image generation is to find the Pareto optimal solution that balances concept preservation and prompt following. Among these 7 models, DreamBooth [] is the best in overall performance, preserving highly detailed visual features while keeping close adherence with text prompt; iii) Currently, personalized T2I models perform well on animal and style category. However, they have not performed well in the human category due to sensitivity to facial details and in the object category due to its diversity. Existing works [, , , , ] focuses on preserving facial features to address the former problem, but the latter has not been extensively explored.

We are presenting DreamBench++ with open-sourced codes and evaluation standardization to promote innovation within the research community.
In addition, we believe our design of the human-aligned & automated evaluation using advanced foundation models is robust and transferrable to other domains and foundation models (e.g., GPT-5 in the future).

## 2 DreamBench++

We introduce DreamBench++, a human-aligned, automated, and diverse benchmark that evaluates the two capabilities of personalized image generation models.
In the following, we describe how we construct DreamBench++ from two aspects:
prompts and data.

Figure: Figure 3: Overall procedure of prompting GPT-4o for automated evaluation. The evaluation instructions are meta-prompting information written by humans, including task description, scoring criteria, scoring range, and format specification. Then, GPT-4o is prompted with reasoning instructions to perform internal thinking that provides a self-aligned task summary and planning. Finally, all prompts and reasoning outputs are joined with image samples for score outputs.
Refer to caption: x3.png

### 2.1 Prompting Multimodal GPT for Automated & Human-Aligned Benchmarking

It is challenging to obtain a solid quantitative understanding of generated models, especially when evaluating visual contents like images that rely on human evaluations [, ].
Thus, it is critical to achieve automated evaluation by utilizing multimodal GPT models, which are trained particularly in the principle of aligning with human preference [, , , ].
This is evidenced by the recent progress achieved by , which demonstrates that GPT-4V [] can serve as a human-aligned text-to-3D generation evaluator.
However, as pointed out by and , multimodal GPT models often fall short in evaluating personalized image generation—often more challenging when distinguishing subtle difference for concept preservation assessment using GPT—still underexplored.
To tackle this issue, we detail how we systematically design the prompt of multimodal GPT (GPT-4o [], by default) for human alignment reinforcement but also improve the reasoning progress that helps the GPT models to be more self-aligned, introduced as follows.

Compare or rate?
There are typically two schemes for quantitatively evaluating generative models in human evaluations: rating and comparison [, ].
The rating scheme requires human reviewers to assign an absolute score to each instance, while the comparison scheme asks human reviewers to express a relative preference among different instances.
Though effective as the comparison scheme is when humans are involved, we find that there are two critical issues.
i) Positional Bias: the scoring results of GPT-4V/GPT-4o is sensitive to the order in which images are presented [, , , , , ], making it unsuitable for comparison scheme.
ii) Quadratic Complexity: As the number of methods increases, the number of essential evaluation runs for numerical rating increases linearly, while the number of comparative assessments increases quadratically.
Therefore, direct numerical rating is more efficient and scalable when evaluating multiple methods.
Hence, in this work, we adhere to the rating scheme, and we establish a 5-level rating scheme where scores are integers ranging from 0 (very poor) to 4 (excellent).

Figure: Figure 4: Dataset construction process of our DreamBench++. We start by obtaining keywords through GPT generation, existing datasets, and human proposals. Next, we collect corresponding images from the internet. These images are then filtered to remove low-quality ones through both model and human assessment. The remaining high-quality images are used as input for GPT-4o to generate text prompts of varying difficulty levels.
Refer to caption: x4.png

Evaluation Instructions
The evaluation instructions serve as the meta-prompting that describes overall tasks, which is shown in Fig. 3.
As stated in Section 1, there are two fundamental quality criteria to be evaluated: i) concept preservation and ii) prompt following.
For each aspect, we use a similar prompt template that contains ❶ task description, ❷ scoring criteria explanation,
❸ scoring range definition, and ❹ format specification. Only the scoring criteria are tailored for different tasks: for concept preservation evaluation, we prompt GPT to focus on shape, color, texture, and facial features (if applicable), while for prompt following evaluation we requested for focus on relevance, accuracy, completeness and context.

Reasoning Instructions
Given the evaluation instructions, it is crucial to reinforce the alignment with both the human instruction and itself to largely leverage the pretrained knowledge.
To this end, we adopt a 2-step evaluation policy as follows:
i) Internal Thinking:
Inspired by Self-Align [], we introduce internal thinking to strengthen task understanding and instruction following capabilities.
Specifically, we prompt the GPT model by asking if it understands the task or not and let it summarize the task.
ii) Summary & Planning: According to the given internal thinking instruction, the GPT will summarize and plan for the evaluation task itself. It can also be viewed as a generalized form of chain-of-thought reasoning [, ].
The complete procedure is illustrated in Fig. 3.

### 2.2 Scaling Up Personalized Image Generation Benchmarking

Pioneering works like DreamBooth [] and SuTI [] have successfully set up baseline datasets for the evaluation of personalized image generation, and DreamBench++ follows them to categorize images into three types: ❶ objects, ❷ living subjects, and ❸ styles.
However, due to the small-scale nature of DreamBench, it is limited as some methods may converge well on its samples while performing unsatisfactorily on other data.
To avoid this possible biased evaluation, we scale up the benchmarking data
by increasing both image numbers and diversity.

Figure: Figure 5: Data distribution visualization. (a) Images comparison between DreamBench and DreamBench++ using t-SNE [, ]. (b) Image and prompt distribution of DreamBench++.
Refer to caption: x5.png

Data Construction from Internet
There are broad images on the Internet, and many datasets are constructed from it [, ].
DreamBench++ mainly collects images from Unsplash [], Rawpixel [], and Google Image Search [], along with contributions from individuals with authorized permissions. Each image’s copyright status has been verified for academic suitability.
As shown in Fig. 4, we collect and construct high-quality data in DreamBench++ by following 3 steps:

- •
Keywords Generation
First, we generate 200 relevant keywords using GPT-4o and join them with the 200 most frequent keywords from Unsplash. After filtering out duplicated keywords, seven human annotators will extend the list to around 300 based on their interests.
- •
Internet Images Collection
Given selected keywords, we retrieved corresponding images from Unsplash, Rawpixel, and Google Image Search. To filter out images unsuitable for personalized image generation, SAM [] is applied to identify subject regions in images and discard those with too small subject areas.
Human annotators will then filter out images with noisy backgrounds.
Curated images were cropped to centralize the subject, resulting in two images per keyword. Keywords that fail to yield suitable images will be discarded in this process.
- •
Prompt Generation
After image collection, 9 text prompts per image were generated using GPT-4o, designed to cover a range of difficulties: 4 prompts for ❶ photorealistic styles, 3 for ❷ non-photorealistic styles, and 2 for ❸ complicated & imaginative contents.
To align with established evaluation methods, we use few-shot prompts selected from PartiPrompts [].
Human calibration ensures that all generated prompts are ethical and without flaws.
As a result, the construction process finally yields 150 high-quality images and 1,350 prompts.

Diversity Visualization
When collecting images, we notice that Internet images possess a bias towards photorealistic styles. To diversify, various non-photorealistic styles are enlisted, and human annotators are tasked to gather images for each style, including anime, sketches, traditional Chinese paintings, artworks, and cartoon characters from games.
Then, a manual selection process ensures a balanced distribution of images across subject classes and between photorealistic and non-photorealistic styles. In Fig. 5(a), we visualize the t-SNE [, ] of images from DreamBench and DreamBench++, which demonstrates the superiority of DreamBench++ in diversity.
Besides, Fig. 5(b) presents the detailed statistics of the image distribution in DreamBench++.

## 3 Experiments

In this section, we provide evidence that DreamBench++ can more accurately reflect the capabilities of existing personalized image generation methods with its diverse and comprehensive data set. Additionally, we demonstrate that our systematic prompts-based GPT is a superior evaluator compared to existing metrics by analyzing their degree of alignment with human ratings.
In order to further facilitate benchmarking research with GPT models, we conduct a comprehensive ablation study to show the designing key points for more human-aligned automated GPT evaluation.

**Table 1: Evaluation of personalized image generation models on DreamBench++. All scores are normalized to 0-1, and -I & -T represent image & text, respectively.**
| Method | T2I Model | Concept Preservation | Prompt Following |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Human | GPT | DINO-I | CLIP-I | Human | GPT | CLIP-T |  |  |
| $\bullet$ Textual Inversion [] | SD v1.5 | 0.316 | 0.378$\pm$0.0012 | 0.437 | 0.726 | 0.604 | 0.624$\pm$0.0033 | 0.302 |
| $\bullet$ DreamBooth [] | SD v1.5 | 0.453 | 0.493$\pm$0.0012 | 0.544 | 0.753 | 0.679 | 0.721$\pm$0.0016 | 0.323 |
| $\bullet$ DreamBooth LoRA [] | SDXL v1.0 | 0.571 | 0.597$\pm$0.0007 | 0.628 | 0.784 | 0.821 | 0.865$\pm$0.0007 | 0.341 |
| $\bullet$ BLIP-Diffusion [] | SD v1.5 | 0.513 | 0.547$\pm$0.0010 | 0.649 | 0.823 | 0.577 | 0.495$\pm$0.0005 | 0.286 |
| $\bullet$ Emu2 [] | SDXL v1.0 | 0.410 | 0.528$\pm$0.0016 | 0.539 | 0.763 | 0.641 | 0.689$\pm$0.0010 | 0.310 |
| $\bullet$ IP-Adapter-Plus ViT-H [] | SDXL v1.0 | 0.755 | 0.833$\pm$0.0008 | 0.834 | 0.917 | 0.541 | 0.413$\pm$0.0005 | 0.282 |
| $\bullet$ IP-Adapter ViT-G [] | SDXL v1.0 | 0.570 | 0.593$\pm$0.0018 | 0.667 | 0.855 | 0.688 | 0.640$\pm$0.0017 | 0.309 |

### 3.1 Experimental Setup

Reimplementation Details
We mainly conduct experiments on the two mainstream methods:
i) $\bullet$ Fine-tuning-based methods, including ❶ Textual Inversion (TI) [], ❷ DreamBooth [], and ❸ DreamBooth LoRA (DreamBooth-L) [, ];
ii) $\bullet$ Encoder-based methods that trains feature adaptation, including ❹ BLIP-Diffusion (BLIP-D) [],
❺ Emu2 [], ❻ IP-Adapter-Plus ViT-H (IP-Adapt.-P) [], and ❼ IP-Adapter ViT-G (IP-Adapt.) [].
All these methods are based on the base T2I models, including SD v1.5 [] and SDXL v1.0 [].
We stay true to the official implementations wherever possible and dedicate significant effort to tuning hyper-parameters to ensure that the performance of each method on DreamBench is consistent with results reported in original papers.
More implementation details can be found in Appendix A.

Figure: Figure 6: Comparison between images of high DINO score and high GPT-4o score. Instances with high human scores are ticked, and those with low human scores are crossed. DINO tends to yield high scores to images that preserve overall shape but do not put much weight on color, texture, and facial features, leading to frequent contradiction with human preference.
Refer to caption: x6.png

Human Annotators
We employ 7 human annotators to score for each instance in DreamBench++ to obtain ground truth human preference data.
We provide human annotators with sufficient training to ensure they fully understand the personalized image generation task and can provide unbiased and discriminating scores. The scoring task and scheme given to humans are identical to those used for GPT, as described in Section 2.
The GPT results and human results are isolated to avoid hindsight bias.
Additionally, we ensure that each instance is
rated by at least two humans to reduce noise.

Figure: Figure 7: A qualitative study of different methods on DreamBench++. We demonstrate the generation quality of different methods on our DreamBench++, including animals, humans, objects, and style, with photo and non-photo-realistic examples. The blue block highlights fine-tuning-based methods, and the green block highlights encoder-based methods. Instances above the dotted line are evaluated for subject preserving, and instances below are evaluated for style preserving.
Refer to caption: x7.png

### 3.2 Main Results

Quantitative Analysis
Table 1 shows the overall evaluation results, including human and GPT-4o rating scores.
From the results, we observe that:
i) DreamBench++ aligns better with humans than DINO or CLIP models.
Driven by our dedicatedly-designed prompts, GPT-4o used by DreamBench++ yields impressive alignment with humans.
This is because humans and DreamBench++ are all advanced in evaluating facial and textural characters and producing scores with a balanced consideration of all aspects.
ii) DINO-I and CLIP-I yield significant divergence from humans in evaluating concept preservation.
This could be because DINO/CLIP scores show a preference for images that preserve shapes or overall styles, as shown in Fig. 6.
For example, methods proficient in posture and detailed texture transformation, such as DreamBooth-L, rank lower than humans and DreamBench++.
iii) Traditional CLIP-T scores are as effective as DreamBench++ in evaluating prompt following, showing strong alignment with humans.

**Table 2: DreamBench++ leaderboard. Both scores for concept preservation (CP) and prompt following (PF) are presented and divided by established concept and prompt categories. The models are ranked by the product of CP and PF scores (CP$\cdot$PF).**
| Method | T2I Model | Concept Preservation | Prompt Following | CP$\cdot$PF |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Animal | Human | Object | Style | Overall | Photorealistic | Style | Imaginative | Overall |  |  |  |
| $\bullet$ DreamBooth LoRA [] | SDXL v1.0 | 0.751 | 0.311 | 0.543 | 0.718 | 0.598 | 0.898 | 0.895 | 0.754 | 0.865 | 0.517 |
| $\bullet$ IP-Adapter ViT-G [] | SDXL v1.0 | 0.667 | 0.558 | 0.504 | 0.752 | 0.593 | 0.743 | 0.632 | 0.446 | 0.640 | 0.380 |
| $\bullet$ Emu2 [] | SDXL v1.0 | 0.670 | 0.546 | 0.447 | 0.454 | 0.528 | 0.732 | 0.719 | 0.560 | 0.690 | 0.364 |
| $\bullet$ DreamBooth [] | SD v1.5 | 0.640 | 0.199 | 0.488 | 0.476 | 0.494 | 0.789 | 0.775 | 0.504 | 0.721 | 0.356 |
| $\bullet$ IP-Adapter-Plus ViT-H [] | SDXL v1.0 | 0.900 | 0.845 | 0.759 | 0.912 | 0.833 | 0.502 | 0.384 | 0.279 | 0.413 | 0.344 |
| $\bullet$ BLIP-Diffusion [] | SD v1.5 | 0.673 | 0.557 | 0.469 | 0.507 | 0.547 | 0.581 | 0.510 | 0.303 | 0.495 | 0.271 |
| $\bullet$ Textual Inversion [] | SD v1.5 | 0.502 | 0.358 | 0.305 | 0.358 | 0.378 | 0.671 | 0.686 | 0.437 | 0.624 | 0.236 |

Qualitative Analysis
Fig. 7 shows selected qualitative results on DreamBench++, which helps provide an intuitive understanding of evaluated models.
With a more comprehensive and diverse collection of images, we have discovered numerous intriguing characteristics of these generation methods that were not apparent on existing datasets such as DreamBench. Specifically, we observe that:
i) Fine-tuning-based methods outperform encoder-based methods on images containing more subject-oriented information, such as an animal or object, as they preserve more intricate details in the generated images. However, for images containing a person, fine-tuning-based methods often fail to preserve facial and clothing features. This suggests that the personalized generation of human images is more demanding for visual concept preservation than for textual following capabilities, which is an advantage for encoder-based methods.
ii) However, for style-oriented cases when subject details are less critical, encoder-based methods perform better than fine-tuning-based methods. This further highlights the strengths of encoder-based methods in that they are more adept at recognizing and extracting high-level visual semantics, including overall shape, style, and thematic features.

Leaderboard
Table 2 shows the detailed results of the leaderboard with respect to the concept and prompt categories defined in Section 2. Note that: i) the human category shows the lowest average score of 0.482, which is -0.204 lower than the highest average score of animal.
This category is very challenging in terms of concept preservation because humans are very sensitive to facial details, and many works are conducted specifically on it [, , , ].
ii) The object is also a relatively difficult category because the characteristics of each same or different object are often very diverse.
In contrast, animals within the same category often share a large degree of visual similarity.
iii) There exists a negative correlation between concept preservation and prompt following. The primary aim of personalized T2I evolution is to identify the Pareto optimum that balances both factors.

**Table 3: Ablation study of prompt design. H, G, D, and C represent Human, GPT-4o, DINO Score, and CLIP Score, respectively. H-H value is also calculated to illustrate human self-alignment.**
| Method | TI [] | DreamBooth [] | DreamBooth-L [] | BLIP-D [] | Emu2 [] | IP-Adapt.-P [] | IP-Adapt. [] |
| --- | --- | --- | --- | --- | --- | --- | --- |
| T2I Model | SD v1.5 | SD v1.5 | SDXL v1.0 | SD v1.5 | SDXL v1.0 | SDXL v1.0 | SDXL v1.0 |
| Concept Preservation |  |  |  |  |  |  |  |
| H-H | 0.685 | 0.647 | 0.656 | 0.613 | 0.746 | 0.602 | 0.591 |
| G-H | 0.544$\pm$0.014 | 0.596$\pm$0.003 | 0.641$\pm$0.007 | 0.362$\pm$0.017 | 0.669$\pm$0.005 | 0.366$\pm$0.017 | 0.458$\pm$0.002 |
| - Internal Thinking | -0.040 | -0.023 | -0.012 | +0.001 | -0.045 | -0.038 | -0.008 |
| - Scoring Criteria | -0.125 | -0.116 | -0.093 | -0.158 | -0.103 | -0.227 | -0.166 |
| - Scoring Range | -0.038 | -0.017 | -0.027 | -0.006 | -0.016 | -0.009 | -0.017 |
| + Human Prior | -0.033 | -0.022 | -0.006 | -0.015 | -0.022 | +0.009 | -0.019 |
| + GPT4V | -0.105 | -0.067 | -0.131 | -0.180 | -0.016 | -0.301 | -0.250 |
| Prompt Following |  |  |  |  |  |  |  |
| H-H | 0.475 | 0.516 | 0.469 | 0.619 | 0.441 | 0.576 | 0.509 |
| G-H | 0.461$\pm$0.007 | 0.506$\pm$0.002 | 0.402$\pm$0.001 | 0.541$\pm$0.003 | 0.422$\pm$0.011 | 0.484$\pm$0.006 | 0.531$\pm$0.006 |
| - Internal Thinking | -0.013 | +0.004 | -0.032 | -0.002 | -0.014 | +0.012 | -0.002 |
| - Scoring Criteria | -0.025 | -0.012 | -0.009 | -0.012 | -0.018 | -0.017 | -0.013 |
| - Scoring Range | -0.010 | -0.013 | -0.011 | +0.043 | -0.038 | +0.060 | +0.036 |
| + GPT4V | -0.010 | +0.012 | 0.000 | -0.111 | -0.007 | -0.161 | -0.134 |

### 3.3 Ablation Study

Table 3 shows the ablation study of the prompt design influences on alignment with humans.
From the results, we observe that:
i) The proposed prompt designs are all necessarily effective, demonstrating the superiority of the prompting method in DreamBench++.
For example, removing the proposed internal thinking leads to a significant drop, indicating the effectiveness of self-alignment for alignment with humans.
ii) The capability of the multimodal GPT used is scalable. This shows that DreamBench++ has the potential to be improved with stronger GPT models in the future.
iii) Some human prior knowledge, such as reminding the GPT not to consider background when assessing visual concept preservation, leads to performance degradation.

**Table 4: The human alignment degree among different evaluation metrics, measured by Krippendorff’s alpha value. H, G, D, and C represent Human, GPT-4o, DINO Score, and CLIP Score, respectively. H-H value is also calculated to illustrate human self-alignment.**
| Method | T2I Model | Concept Preservation $\bm{\mathrm{Kd}_{\bar{O}}}$ | Prompt Following $\bm{\mathrm{Kd}_{\bar{O}}}$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| H-H | G-H | D-H | C-H | H-H | G-H | C-H |  |  |
| $\bullet$ Textual Inversion [] | SD v1.5 | 0.685 | 0.544$\pm$0.014 | 0.262 | -0.030 | 0.475 | 0.461$\pm$0.007 | 0.267 |
| $\bullet$ DreamBooth [] | SD v1.5 | 0.647 | 0.596$\pm$0.003 | 0.408 | 0.229 | 0.516 | 0.506$\pm$0.002 | 0.185 |
| $\bullet$ DreamBooth LoRA [] | SDXL v1.0 | 0.656 | 0.641$\pm$0.007 | 0.371 | 0.321 | 0.469 | 0.402$\pm$0.001 | 0.022 |
| $\bullet$ BLIP-Diffusion [] | SD v1.5 | 0.613 | 0.362$\pm$0.017 | -0.078 | -0.186 | 0.619 | 0.541$\pm$0.003 | 0.319 |
| $\bullet$ Emu2 [] | SDXL v1.0 | 0.746 | 0.669$\pm$0.005 | 0.518 | 0.258 | 0.441 | 0.422$\pm$0.011 | 0.230 |
| $\bullet$ IP-Adapter-Plus ViT-H [] | SDXL v1.0 | 0.602 | 0.366$\pm$0.017 | -0.141 | -0.150 | 0.576 | 0.484$\pm$0.006 | 0.256 |
| $\bullet$ IP-Adapter ViT-G [] | SDXL v1.0 | 0.591 | 0.458$\pm$0.002 | -0.073 | -0.212 | 0.509 | 0.531$\pm$0.006 | 0.196 |
| $\bm{\mathrm{Ratio}_{\bar{O}}}$ |  | 100% | 79.64% | 25.54% | 3.34% | 100% | 93.18% | 42.48% |

## 4 Discussions

### 4.1 Is DreamBench++ aligned with humans?

Table 4 shows a more rigorous study of human alignment level using the mean Krippendorff’s alpha value [].
The results show that DreamBench++ is a highly human-aligned benchmark.
Notably, DreamBench++ achieves 79.64% and 93.18% evaluation consistency with human’s evaluation in concept preservation and prompt following capabilities, respectively. This result is +54.1% and +50.7% higher than traditional DINO and CLIP metrics.

**Table 5: DreamBench [] and DreamBench++ results comparison with traditional metrics. ^∗Unlike DreamBench, DreamBench++ uses one single reference image per instance; thus, the training steps and learning rate of $\bullet$ fine-tuning-based methods are slightly reduced to avoid overfitting.**
| Method | T2I Model | DreamBench [] | DreamBench++ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DINO-I | CLIP-I | CLIP-T | DINO-I | CLIP-I | CLIP-T |  |  |
| $\bullet$ Textual Inversion^∗ [] | SD v1.5 | 0.557 | 0.753 | 0.259 | 0.437 | 0.726 | 0.302 |
| $\bullet$ DreamBooth^∗ [] | SD v1.5 | 0.678 | 0.786 | 0.301 | 0.544 | 0.753 | 0.323 |
| $\bullet$ DreamBooth LoRA^∗ [] | SDXL v1.0 | 0.646 | 0.769 | 0.325 | 0.628 | 0.784 | 0.341 |
| $\bullet$ BLIP-Diffusion [] | SD v1.5 | 0.630 | 0.784 | 0.293 | 0.649 | 0.823 | 0.286 |
| $\bullet$ Emu2 [] | SDXL v1.0 | 0.753 | 0.842 | 0.283 | 0.539 | 0.763 | 0.310 |
| $\bullet$ IP-Adapter-Plus ViT-H [] | SDXL v1.0 | 0.846 | 0.902 | 0.272 | 0.834 | 0.917 | 0.282 |
| $\bullet$ IP-Adapter ViT-G [] | SDXL v1.0 | 0.681 | 0.835 | 0.295 | 0.667 | 0.855 | 0.309 |

### 4.2 Is data diversity necessary?

To investigate the necessity of diverse and rich evaluation data,
we compare the results on DreamBench and DreamBench++ using DINO and CLIP metrics.
Table 5 demonstrates that diverse data in DreamBench++ is necessary to avoid biased evaluation.
Although the results are overall consistent, especially for encoder-based methods, fine-tuning-based methods such as TI and DreamBooth both show a notable score drop.
We argue that this is due to the reduced training steps and learning rate limited by the single-image fine-tuning, which may make the model not fully converged.
However, unlike other encoder-based methods, Emu2 shows a significant score drop.
Emu2 performs well only on natural images and simple text, while it cannot generate valid results when texts are complex or stylized or with anime reference images, as shown in Fig. 8.

Figure: Figure 8: Case study of successful and failure case on DreamBench++ with Emu2.
Refer to caption: x8.png

**Table 6: Study of Chain-of-Though (CoT) and In-context Learning (ICL) on human alignment.**
| Method | TI [] | DreamBooth [] | DreamBooth-L [] | BLIP-D [] | Emu2 [] | IP-Adapt.-P [] | IP-Adapt. [] |
| --- | --- | --- | --- | --- | --- | --- | --- |
| T2I Model | SD v1.5 | SD v1.5 | SDXL v1.0 | SD v1.5 | SDXL v1.0 | SDXL v1.0 | SDXL v1.0 |
| H-H | 0.685 | 0.647 | 0.656 | 0.613 | 0.746 | 0.602 | 0.591 |
| w/o CoT & w/o ICL | 0.544 | 0.596 | 0.641 | 0.362 | 0.669 | 0.366 | 0.458 |
| + 1 shot ICL | -0.046 | -0.019 | -0.043 | +0.013 | -0.028 | -0.098 | -0.066 |
| + 2 shot ICL | -0.042 | -0.023 | -0.022 | -0.033 | -0.036 | -0.085 | -0.054 |
| w/ CoT & w/o ICL | 0.510 | 0.576 | 0.602 | 0.329 | 0.644 | 0.359 | 0.418 |
| + 1 shot ICL | -0.040 | -0.008 | -0.009 | -0.020 | -0.035 | -0.145 | -0.086 |
| + 2 shot ICL | -0.030 | -0.002 | -0.002 | -0.051 | -0.031 | -0.155 | -0.082 |

### 4.3 Can we use free lunch to improve DreamBench++ evaluation?

Table 6 shows the result of utilizing free lunch techniques, including chain-of-thought (CoT) [] and In-Context Learning (ICL) [, ]. CoT indicates that GPT-4 articulates its reasoning process before scoring, and ICL indicates GPT-4o is provided with human-written few-shot examples.

Chain-of-Thought:
i) CoT is effective in evaluating prompt following capability.
Through CoT, the model more accurately discerns the significance of phrases such as “morphs into a mythical dragon”, allowing it to assign a more appropriate evaluation score.
ii) CoT does not bring improvement in concept preservation evaluation.
We argue that CoT may shift attention to unnecessarily important background or texture information, as shown in Fig. 9.

Figure: Figure 9: Case study on CoT Prompting. We find that (a) CoT prompting can improve text following evaluation by recognizing important parts of the prompt. (b) However, it may also hinder visual concept preservation by drifting GPT’s attention away from the subject.
Refer to caption: x9.png

In-Context Learning: ICL counterintuitively leads to a drop in alignment. This could be attributed to the patching scheme, sample selection, or inherent bias within GPT-4o, making it non-trivial to prompt effectively.
Thus, we provide our detailed prompt and hope to inspire future works.

## 5 Related Works

Personalized Image Generation aims to preserve concept consistency while accommodating the diverse contexts suggested by the instructions.
In general, it can be traced back to early efforts on pixel-to-pixel (Pix2Pix) translation where the personalization orientation is free-form texts [, , ] or predefined translation across styles, seasons, species, or plants, etc [, , , , ].
Modern efforts go beyond Pix2Pix translation toward a free-form image generation conditioned on both reference images and prompts.
Some works focus on fine-tuning techniques that turn a general T2I model into a specialist personalization model [, , , , ] using LoRA [] or contrastive learning [, ], learning the subject or style information by reconstructive autoencoding [, ].
However, the necessity to fine-tune for new subjects limits their scalability.
In contrast, encoder-based methods can generate subject-guided or style-guided images or edit images following prompts with one shot.
Encoder- or adapter-based methods [, , , , , , , , ] train an encoder to encode the conditional image into embeddings, which are integrated into cross-attention mechanism in the diffusion process [, , , ]. Adapter-free methods [, , , , ] extract the information, such as attention maps [, RBModulation24] from reference images, and fuse them into the image generation process.
Furthermore, multimodal large language models (MLLMs) that are trained on extensive multimodal sequences can also serve as general foundation models [, , , ].

Benchmarking Image Generation involves a variety of metrics that focus on different aspects. Inception Score [] and FID [] judge image quality, while LPIPS [], DreamSim [], CLIP-I [], and DINO Score [] measure perceptual similarity. In text-guided generation, prompt-image alignment can be assessed by CLIP-T [], CLIPScore [], and BLIP Score [, ].
However, these metrics often fall short of reflecting human perception.
To address this, human-aligned metrics [, , ] have been introduced, offering a more perceptive evaluation. Yet, they face limitations in scaling with the pace of new model developments. Thus, the necessity for automated and sustainable evaluation methods has emerged, with some [, , ] leveraging reward-model-based methods to encode human preferences, while others [, , , , , ] use multimodal  [, , , ] to automate the process and better mirror human tastes. While MLLM-based methods show promise in aligning with human preferences [, , , ], automated personalized evaluation remains an unresolved issue.
VIEScore [] assesses image generation quality by prompting GPT-4V [] and LLaVA [, ], but is limited to four models in subject-driven tasks and obtains suboptimal results. Meanwhile, Dreambench [], a common benchmark for personalized generative evaluation, only consists of 30 simple objects and lacks diversity comprehensiveness.

## 6 Conclusions

This paper introduces DreamBench++, a human-aligned personalized image generation benchmark.
Extensive and comprehensive experiments have demonstrated our advantages in dataset diversity and complexity, as well as the alignment of our evaluation metrics with human preferences.
In addition, we offer insights into prompt design for advanced multimodal GPTs, emphasizing the potential and challenges of enhancing GPT evaluation through chain-of-thought prompting and in-context learning. Our work aims to support future research on personalized image generation by providing a human-aligned benchmark and heuristics in utilizing advanced multimodal GPTs in visual evaluation.

## Appendix A Implementation Details

**Table 7: Training hyperparameters on DreamBench and DreamBench++. BS: batch size, LR: learning rate, Steps: training steps.**
|  |  |  | DreamBench |  |  |  | DreamBench++ |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | T2I Model | BS | LR | Steps |  | BS | LR | Steps |
| Textual Inversion [] | SD v1.5 | 4 | 5e-4 | 3000 |  | 1 | 5e-4 | 3000 |
| Dreambooth [] | SD v1.5 | 1 | 2.5e-6 | 1000 |  | 1 | 2.5e-6 | 250 |
| Dreambooth LoRA [] | SDXL v1.0 | 4 | 5e-5 | 500 |  | 1 | 5e-5 | 500 |

The configurations for the training hyperparameters used in training-based methods on DreamBench and DreamBench++, are detailed in Table 7. During the inference stage, all methods employ a guidance_scale of 7.5 and execute 100 inference steps, with the exception of Emu2, which uses a guidance_scale of 3 and performs 50 inference steps. Furthermore, BLIP-Diffusion and IP-Adapter incorporate negative prompts, as demonstrated in Table 8. Specifically, IP-Adapter includes an additional parameter, ip_adapter_scale, set at 0.6.

**Table 8: Negative Prompt Templates**
| Method | T2I Model | Negative Prompt |
| --- | --- | --- |
| BLIP-Diffusion [] | SD v1.5 | over-exposure, under-exposure, saturated, duplicate, out of frame, lowres, cropped, |
| worst quality, low quality, jpeg artifacts, morbid, mutilated, ugly, bad anatomy, |  |  |
| bad proportions, deformed, blurry, duplicate |  |  |
| IP-Adapter ViT-G [] | SDXL v1.0 | deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality |
| IP-Adapter-Plus ViT-H [] | SDXL v1.0 | deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality |

We dedicate significant effort to tuning hyperparameters to ensure that the performance of each method on DreamBench is consistent with results reported in original papers.
As shown in Table 9, the results of our reproduction are comparable to or even better than the official results reported by the original papers.

**Table 9: Reproduced results. Our reproduction is comparable to or better than the official results. N/A denotes that the official paper does not report the corresponding results.**
| Method | T2I Model | DINO-I | CLIP-I | CLIP-T |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Official | Reproduction | Official | Reproduction | Official | Reproduction |  |  |
| $\bullet$ Textual Inversion [] | SD v1.5 | 0.569 | 0.557 | 0.780 | 0.753 | 0.255 | 0.259 |
| $\bullet$ DreamBooth [] | SD v1.5 | 0.688 | 0.678 | 0.803 | 0.786 | 0.305 | 0.301 |
| $\bullet$ DreamBooth LoRA [] | SDXL v1.0 | N/A | 0.646 | N/A | 0.769 | N/A | 0.325 |
| $\bullet$ BLIP-Diffusion [] | SD v1.5 | 0.594 | 0.630 | 0.779 | 0.784 | 0.300 | 0.293 |
| $\bullet$ Emu2 [] | SDXL v1.0 | 0.766 | 0.753 | 0.850 | 0.842 | 0.287 | 0.283 |
| $\bullet$ IP-Adapter-Plus ViT-H [] | SDXL v1.0 | N/A | 0.846 | N/A | 0.902 | N/A | 0.272 |
| $\bullet$ IP-Adapter ViT-G [] | SDXL v1.0 | N/A | 0.681 | N/A | 0.835 | N/A | 0.295 |

## Appendix B Limitation & Future Work

Human-aligned evaluation & benchmarking is an emerging but challenging direction, and we have only made preliminary attempts at personalized image generation. Moreover, our evaluation results heavily rely on the advancements of multimodal large language models and require carefully designed system prompts. We believe that as visual world models continue to develop, the evaluation performance will be further optimized. Our future work will focus on more applications with human-aligned evaluation, such as 3D generation [, , , ], video generation [, , ], autonomous driving [, , ], and even embodied visual intelligence [, , , , , ].

## Broader Impact

Powerful as the T2I generative models pretrained on large-scale web-scraped data, the models may be misused as illegal or unethical tools for generating NSFW content.
This potential impact can also be brought by personalized T2I models as they are typically built on the pretrained T2I foundation models.
As a result, it is critical to use tools such as NSFW detectors to avoid such content during both usage and evaluation.
For example, the data used for evaluation must avoid the NSFW content by data filtering.
In this paper, such contents are filtered out by human annotators.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you do specific comparisons between DREAMBENCH++ and existing benchmarks like DALL-EVAL and HRS-Bench. For example, a comparison table showing the number of samples, types of skills assessed, and evaluation metrics used in each benchmark is necessary. This would help clarify the unique contributions of DREAMBENCH++.

2. May I ask if the authors have considered developing new evaluation metrics, proposing improvements to existing personalized T2I models based on benchmark insights?
soundness: 3
strengths: I appreciate that the authors engage a lot of effort to build a new benchmark, including design prompt, collecting images, and conducting experiments among several existing T2I models. The paper is written well and clear, the figure is informative, which is good.
summary: This paper proposed a new T2I evaluation benchmark, DREAMBENCH++, which is introduced as a human-aligned benchmark for personalized image generation, addressing the limitations of current evaluations that are either misaligned with human judgment or time-consuming and expensive. The researchers systematically design prompts to make GPT models both human-aligned and self-aligned, enhanced with task reinforcement, and construct a comprehensive dataset of diverse images and prompts. By benchmarking 7 modern generative models, this paper demonstrates that DREAMBENCH++ achieves significantly more human-aligned evaluation, contributing to innovative findings in the field.
weaknesses: 1. I am confused about the motivation of building this new benchmark actually. As stated on line 084-085, "can we comprehensively evaluate these models to figure out which technical route is superior and where to head?", it's hard for reviewers to understand what distinguishes your benchmark compared with other existing benchmarks. Does DREAMBENCH++ assess more skills than existing T2I benchmarks? Does DREAMBENCH++ include more samples or higher quality images? A comparison table (DREAMBENCH++ v.s. existing benchmarks) is necessary to convince reviewers how the new benchmark can benefit T2I community.

2. Nowadays, there are several existing T2I benchmarks, for example, DALL-EVAL [1] and HRS-Bench [2]. So a new benchmark alone may not contribute enough in this field. It would be good to contribute one technical contribution to somehow address the challenge which is emphasized in the newly built benchmark.

[1] Cho, J., Zala, A., & Bansal, M. (2023). Dall-eval: Probing the reasoning skills and social biases of text-to-image generation models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3043-3054).
[2] Bakr, E. M., Sun, P., Shen, X., Khan, F. F., Li, L. E., & Elhoseiny, M. (2023). Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20041-20053).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: How are the numbers calculated? Can you specify how you used Krippendorff’s alpha value? And where does the number of 54.1% and 50.7% come from? I noticed that the CLIP and DINO scores fairly correlate with human scores.
soundness: 2
strengths: It fills a gap in human-aligned and automated evaluation of the personalized text-to-image generation task by introducing LLMs. The dataset and evaluation metric is clearly described in detail.
summary: This work introduces a new benchmark for personalized text-to-image generation, including a dataset with a greater number of images and prompts compared to DreamBench, and an automatic evaluation method that assesses two key aspects of the task: (i) concept preservation and (ii) prompt following, using multimodal large language models like GPT. This new evaluation method addresses the limitations of previous DINO and CLIP and achieves higher human alignment.
weaknesses: 1. The new dataset aims to include a larger variety of images but the source of these images is still relatively narrow (only 3 websites). The new dataset can still be biased due to the preference of these websites, and there is no sufficient evidence to show its diversity (only the visualization of t-SNE).
2. It is claimed that the method is transferable to other foundation models. It is not straightforward because the method is specifically designed for GPT4 and no experiment showed its transferability.
3. The dataset only contains 1 image for every instance. Although multiple images are claimed to be unnecessary, the results in Fig. 9 show serious overfitting when using only 1 reference image.
4. There are other key aspects that need evaluation. For example, a common problem of fine-tuning-based methods is overfitting. We generally don't want the generated images to be too similar to the reference image except for the identity of the object. It is worth trying to evaluate the overfitting problem using GPT.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness part. In addition,

- Why is the comparison scheme considered unsuitable in lines 198-199 if the scoring results are sensitive? Was "comparing" mistakenly written as "scoring"?

- Stable Diffusion often generates noisy backgrounds effectively and frequently. Why did the authors judge it as unsuitable for personalized generation and remove it?

- Are the example prompts shown in Figure 4 intended for personalized editing with those specific prompts?

- In Table 1, the highest-scoring model is consistent across Human, GPT, DINO-I, CLIP-I, and CLIP-T. Can DREAMBENCH++ really be said to be more aligned with humans than DINO or CLIP?

- Contrary to the authors' claim, Figure 6 does not clearly show that DINO or CLIP prefer shape and overall styles. Also, do the authors believe that favoring shape and overall styles is not aligned with human preference?
soundness: 3
strengths: This paper proposes a metric that closely aligns with human preferences for evaluating personalized generated images and introduces a suitable benchmark. Compared to existing benchmarks, it constructs a much more diverse and extensive dataset and demonstrates evaluation results that are better aligned with humans than CLIP and DINO.
summary: The paper presents DREAMBENCH++, a human-aligned benchmark for evaluating personalized image generation models using advanced multimodal models like GPT-4o for automated, human-like assessments. It addresses the misalignment and cost issues of traditional evaluation methods by focusing on prompt following and concept preservation with a diverse dataset. DREAMBENCH++ demonstrates superior alignment with human evaluations, offering a comprehensive and unbiased framework to advance personalized image generation research.
weaknesses: - For text-to-image generation models, there are many metrics, including TIFA[1], that improve upon CLIP. It is necessary to demonstrate alignment with humans not only compared to CLIP and DINO but also in comparison with these existing studies.

[1] Hu, Yushi, et al. "Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

- The prompts used for GPT-4o are abstract. Studies that evaluate generated images using VQA typically ask absolute questions, such as the presence, location, or number of objects, to fully leverage GPT-4o’s performance. However, the prompting approach in this paper treats GPT-4o like a human, and the justification relies only on citations from other studies. In practice, some degree of hallucination occurs when GPT-4o evaluates images, which the paper overlooks, making this a significant drawback.

-The scoring system from 0 to 4 is ambiguous. Some images may have around 10 objects, with 1 or 2 that do not match, while others may have all objects correct but differ in style. What are their scores? While GPT-4o might provide consistent evaluations, humans are less likely to be consistent. To address this, a larger user study should have been conducted, but only seven people participated, with each instance evaluated by at least two annotators. This means some images were reviewed by just two people, making the number of annotators insufficient.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Q1) Any design rationale on the score scale? Why not use a score in scale from 0 to 1? This would be more aligned with the range of traditional metrics (e.g. CLIPScore, LPIPS, DINO etc.)?

Q2) How do the authors verify that the automated evaluation with GPT results are making sense (such that the reasoning fully reflects the score)?

Q3) Please share the analysis of the costs and time needed for the human annotation, and how many data instance were annotated in total. Will the human annotated data be released?  

Q4) Authors might want to consider adding more models for comparison, including DisenBooth(ICLR 2024), EZIGEN (ArXiv 2024), 

---
I believe this study will interest a broad audience. However, the contribution feels somewhat limited, and the discussion does not fully align with the claimed goal in the introduction. If W1 and W2 can be addressed, I would likely consider raising my score.
soundness: 3
strengths: S1) Fruitful discussion in diversity study, showcasing that models are good at certain types of generation. (e.g. all models generally perform better in Animal and Style due to sensitivity to facial details and diverse object categories). This aligns with the goal to figure out which model is superior in certain types of generation.

S2) Well-organized study in showcasing the impact of prompt design including COT, ICL, Internal thinking etc.. when comes to automated evaluation with GPT.
summary: This paper contributed a benchmark and a dataset for evaluating personalized image generation task, also an automated evaluating metric using GPT to address the issues of cost inefficient human evaluation, and also the lack of diversity in previous dataset/benchmark works. It included 7 models ( 3 more than prior works), and developed an semi-automated dataset creation pipeline. It also expanded dataset to 5x of images and 54x of prompts, and proposed the use of automated evaluating metric (GPT), with different settings studied (e.g. COT and ICL). The main goal of this work is to comprehensively evaluate these models to figure out which technical route is superior and where to head.
weaknesses: W1) In Section 3.2 the authors claimed that "Table 1 results show that DreamBench++ aligns better with humans than DINO or CLIP models...", which is not very convincing. While it makes sense to show that the ranking order from GPT is more correlated to Human compared to DINO-I and CLIP-I, It is expected to show the correlation (Spearman / Pearson). I would suggest the authors to add a table regarding correlation between Human and GPT, and between Human and DINO/CLIP, supporting the claim. Also It seems a bit odd to use Krippendorff's Alpha to compare human ratings are other rating (GPT/DINO/CLIP) in Table 4. The two scales are inherently different and the meaning of ratings are different. Spearman / Pearson correlation would be a better metric for cross-scale reliability. Krippendorff's Alpha would be suitable for H-H as showing the inter-rater reliability.

W2) The paper would have been much stronger if the authors included a section to discuss which model performs the best in certain types of generation and how is it associated with certain technical route. This will be more aligned to the purposed goal "figure out which technical route is superior and where to head."

W3) Minor confusion when reading the table 1 and 3 directly. For example, Table 3 does not mention what exactly are the score values are. Please extend table captions to explain what are the values in the revised version.

W4) Please highlight the best model in each categories from Table 2 in the revised version. Maybe a leaderboard showcasing the best models in each category.

Meta-review: The paper introduces DreamBench++, a new benchmark dataset and evaluation metrics for personalized image generation. The authors propose a GPT-based automated evaluation that better aligns with human judgments. The benchmark contains a diverse set of images and prompts, and the authors analyze seven different personalized image generation methods, showing that the proposed metrics correlate more closely with human ratings than commonly used DINO and CLIP.

Strength
* The contribution of a new benchmark dataset and accompanying evaluation metrics is widely appreciated.
* The paper provides abundant analysis and experiments, showing valuable insights into the performance of various personalized image generation methods.

Weakness
* The necessity of the new benchmark compared to existing ones is not convincingly justified.
* The claimed diversity of the dataset and the generalizability of the GPT-based evaluation method are not sufficiently supported. Reviewers questioned potential biases in data source and the lack of evidence for the method’s transferability across different models.
* The authors claim that a single reference image is sufficient, but this is not well justified.
* The evaluation metrics do not cover all crucial aspects, such as overfitting.
* While the paper compares its approach mainly to DINO and CLIP, it lacks thorough comparisons with more recent or metrics (e.g., TIFA), and promised comparisons were not presented in the rebuttal.
* The reliance on GPT is not fully justified, and the scale of human annotations may be too limited for drawing robust conclusions.

Despite the concerns raised, the reviewers acknowledge the value of the new dataset and automated evaluation metrics presented in the paper. However, the contribution would be stronger if the authors: (1) provide a robust justification for the generalizability of the proposed metrics across various dimensions, (2) justify the design choice of using a single reference image or extend the benchmark to incorporate multiple references, (3) ensure the coverage of the proposed metrics, and (4) demonstrate that the scale and quality of human annotations are sufficient for reliable conclusions. Nevertheless, the paper is considered to meet the acceptance criteria if the authors incorporate the clarifications and additional materials provided in the rebuttal.

Decision: Accept (Poster)