Title: DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation
Venue: NeurIPS 2025
Accepted: no
Keywords: Subject-driven text-to-image generation

Benchmark

Hierarchical sampling

Difficulty and scenario classification

Human-aligned evaluation

Subject Identity Consistency Score (SICS)
Abstract: Significant progress has been achieved in subject-driven text-to-image (T2I) generation, which aims to synthesize new images depicting target subjects according to user instructions. However, evaluating these models remains a significant challenge. Existing benchmarks exhibit critical limitations: 1) insufficient diversity and comprehensiveness in subject images, and 2) inadequate granularity in assessing model performance across different subject difficulty levels and prompt scenarios. To address these limitations, we propose DSH-Bench, a comprehensive benchmark that enables systematic multi-perspective analysis of subject-driven T2I models through three principal innovations: 1) a hierarchical taxonomy sampling mechanism ensuring comprehensive subject representation across 58 fine-grained categories, 2) an innovative classification scheme categorizing both subject difficulty level and prompt scenario for granular model capability assessment, and 3) a novel Subject Identity Consistency Score (SICS) metric demonstrating 9.4\% higher correlation with human evaluation compared to existing measures in quantifying subject preservation. Through empirical evaluation of 15 subject-driven T2I models, DSH-Bench uncovers previously obscured limitations in current approaches while establishing concrete directions for future research.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DSH-Bench, a new benchmark for subject-driven text-to-image (T2I) generation, addressing critical gaps in diversity and evaluation granularity present in prior benchmarks. The authors propose a hierarchical taxonomy-based approach to dataset construction, ensuring broad and fine-grained subject coverage (58 categories, 459 subjects). Prompts are systematically generated and classified by six scenario types and three subject difficulty levels. A novel, human-aligned evaluation metric—Subject Identity Consistency Score (SICS)—is introduced, leveraging fine-tuned vision-language models for more accurate and efficient subject preservation assessment. Empirical evaluations of 15 SOTA models using DSH-Bench uncover new insights into current limitations and trade-offs in subject-driven T2I generation, especially with respect to subject complexity and prompt scenario robustness. The benchmark, data, code, and metric will be released publicly.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- **Comprehensive Dataset Construction:**  \n  The hierarchical taxonomy ensures much greater subject diversity and coverage than previous works, substantially reducing evaluation bias.\n- **Granular, Scenario-Aware Evaluation:**  \n  Categorizing both subjects and prompts by difficulty and scenario allows for detailed diagnosis of model weaknesses, going beyond simple aggregate metrics.\n- **Novel, Human-Aligned Metric (SICS):**  \n  SICS correlates better with human judgments than existing automated metrics (CLIP, DINO, GPT-4o), while being more efficient and scalable.\n- **Extensive Empirical Evaluation:**  \n  Fifteen leading models are systematically compared, and analysis reveals nuanced failure modes not captured by previous benchmarks.\n\n**Weaknesses**\n- **Limited Generalization Evidence for SICS:**  \n  While SICS is shown to outperform baselines on the constructed dataset, its robustness to new subject domains or out-of-distribution data is not deeply examined.\n- **Prompt/Scenario Design:**  \n  Although prompt scenarios are well-motivated, the generation process relies heavily on GPT-4o, which may introduce subtle biases or artifacts. More details on prompt validation or diversity analysis would strengthen the work.\n\nQUESTIONS:\n1. **Generalization of SICS:**  \n   How does SICS perform on subject domains outside those used for training (e.g., medical or satellite imagery)? Score would increase if cross-domain generalization is empirically validated.\n\n2. **Potential single model bias?**  \n   Potential single-model bias with GPT-4o only in classification -- I would at least double check on a sample (say ~100 images) with a set of  candidates at or above the performance tier of 4o: Claude, Gemini 2.5 Pro, etc.\n\nLIMITATIONS:\nPrompts manually reviewed (3.1.3). Ideally the authors could also confirm the 459 images are also reviewed along the lines, as this is a reasonably manageable scale for human review.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DSH-Bench, a systematically annotated benchmark for subject-driven text-to-image generation that expands prior datasets from 6–15 to 58 fine-grained categories, assigns each of its 459 reference images a difficulty level (easy/medium/hard) and six scenario tags (background change, viewpoint/size, interaction, attribute, style, imagination), and pairs them with 5 508 GPT-4o-crafted prompts. To evaluate subject preservation efficiently, the authors fine-tune a 7B vision-language model into the Subject Identity Consistency Score (SICS), which aligns more closely with human judgments than GPT-4o while greatly reducing evaluation cost. They benchmark 15 personalization methods across three axe: subject identity, prompt adherence, and image quality—providing diagnostic analyses by category, difficulty, and scenario, and show results on previous open-source pipelines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n(1) The scale of the benchmark is large compared with previous benchmark for image customization like DreamBench and DreamBench++.\n(2) The authors report their performances on various baseline image customization methods, which makes this benchmark paper pretty solid.\n(3) The authors classify the objects into different difficulty levels, and propose human-aligned and more efficient metric, which contributes to the community.\n\nWeaknesses:\nThe evaluation of the baselines on multi-subject customization task is missing. A good benchmark dataset should also support evaluation on multi-subject customization. Would like to see the evaluation results in the rebuttal.\n\nQUESTIONS:\n(1) How is the hard level of the subjects divided? Is it divided based on experiments?\n(2) Could you classify the dataset based on categories? I think it is quite useful for the community.\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on subject-driven text-to-image benchmark. Authors propose DSH-Bench, addressing key shortcomings of existing benchmarks: 1) limited subject diversity and coarse evaluation granularity—DSH-Bench provides a hierarchical subject taxonomy (58 categories), 2) a classification scheme for subject difficulty and prompt type, and 3) a new Subject Identity Consistency Score (SICS) that aligns more closely with human judgments. By evaluating 15 T2I models, authors reveal limitations in current methods and outlines future research directions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The authors introduce a new benchmark for subject-driven text-to-image (T2I) generation, addressing a research gap by incorporating a hierarchical evaluation based on subject difficulty levels. This benchmark has the potential to yield valuable insights for the community.\n\n2. The paper presents extensive experiments using the proposed benchmark across 15 T2I models. Additional detailed analyses are included in the supplementary material. These thorough evaluations enhance the paper’s credibility and provide meaningful insights.\n\nWeaknesses:\n\n1. The clarity of the paper could be improved. A comparison table summarizing the differences between this and existing T2I benchmarks would be helpful—such as comparing the number of images and prompts, and explicitly highlighting the focus on subject difficulty classification. This would better illustrate the specific research gap the benchmark aims to fill.\n\n2. Some experimental findings are somewhat expected. For instance, the observation in line 274 that \"current subject-driven T2I models exhibit performance degradation on hard-level subjects\" is intuitive. The analysis could be deepened by highlighting findings that existing benchmarks cannot reveal—for example, conducting a per-category analysis. Are there specific subjects that consistently perform well or poorly across all difficulty levels? This would provide more nuanced insights into model behavior.\n\nQUESTIONS:\nPlease refer to weakness part.\n\nLIMITATIONS:\nAuthors already have discussed some limitations in their paper. Please check my concerns in weakness part.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DSH-Bench, a new benchmark for evaluating subject-driven text-to-image (T2I) generation models. The benchmark features a hierarchical taxonomy of subject images, a classification scheme for subject difficulty and prompt scenarios, and a new metric for subject identity consistency (SICS). The authors evaluate 15 subject-driven T2I models on DSH-Bench and claim that it uncovers previously obscured limitations in current approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-  The hierarchical taxonomy is well-defined and leads to a clearly more diverse dataset than existing T2I evaluation benchmarks.\n-  The benchmark includes a relatively large and diverse set of subject images and prompts.\n-  The authors evaluate 15 SOTA T2I methods across multiple dimensions: subject preservation, prompt following, and image quality.\n\nWeaknesses\n-  The primary contribution of this work is the construction of a more comprehensive benchmark for evaluating subject-driven T2I generation. As such, it is more appropriate for the NeurIPS Datasets and Benchmarks Track rather than the main conference track, which emphasizes algorithmic novelty.\n-  The benchmark relies heavily on human annotations, yet the paper omits critical details about the subjective evaluation process—such as the participant demographics, evaluation protocol, and statistical reliability (e.g., inter-rater agreement) of the collected annotations.\n-  The proposed SICS metric is essentially a fine-tuned Qwen2.5-VL-7B model trained on human-annotated labels. This represents more of an engineering effort than a substantive research contribution. While the authors claim a 9.4% improvement over GPT-4o in Kendall’s τ, the paper lacks qualitative examples or error analysis to illustrate where SICS succeeds or fails compared to prior metrics.\n-  The core components of DSH-Bench—namely the hierarchical taxonomy, difficulty-level classification, and evaluation via fine-tuned LLMs—are incremental extensions of techniques already established in prior benchmarks (e.g., DreamBench++). The novelty in methodology is thus limited.\n-  The paper does not clearly explain how the SICS metric is computed or how it substantially differs from existing evaluation methods. Furthermore, SICS is trained on a relatively small dataset (5,000 image-text pairs), which may limit its ability to generalize across the full spectrum of subject appearance variations.\n-  The presentation of experimental results is not intuitive. For instance, from the tables provided, it is difficult to directly discern which T2I methods are top-performing or underperforming across different evaluation dimensions.\n\nQUESTIONS:\nSee the Weaknesses part above for questions.\nThe paper contains many typos, e.g., “Meidum” in Figure 6(a) should be corrected to “Medium”.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}